[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4747/ -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 07 Aug 2019 05:27:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. IMPALA-8781: Fix TestResultSpooling::test_multi_batches Prefix the query in TestResultSpooling::test_multi_batches with the database name. This was causing the Dockerized tests to fail. I double checked what other tests do and all the ones I saw either switch to the appropriate database or prefix the table name using the database name. The latter seemed more straightforward. I was not able to re-produce this locally, and its odd that this only affected the Dockerized tests (even more odd is that it seems to either be intermittent, or only affecting Dockerized tests triggered by gerrit-verify-dryrun-external). Regardless, it is a straightforward fix that makes the TestResultSpooling::test_multi_batches consistent with the rest of the tests. Testing: * Ran test_result_spooling.py locally using both bin/impala-py.test and tests/run-tests.py. Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Reviewed-on: http://gerrit.cloudera.org:8080/14022 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_result_spooling.py 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 04:06:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 03:38:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() After raising the maximum bit width for bit packing to 64 bits, DictDecoder accepted bit widths between 32 and 64, but internally it uses 32 bit integers and unpacking ran into a DCHECK. Adding a check to DictDecoder to catch if the bit width is higher than 32. Testing: Added a test that asserts that DictDecoder accepts bit widths 0-32 and rejects higher bit widths which could still be unpacked otherwise. Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Reviewed-on: http://gerrit.cloudera.org:8080/14019 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/util/dict-encoding.h M be/src/util/dict-test.cc 2 files changed, 31 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/436/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 02:56:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Hello Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14021 to look at the new patch set (#4). Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala - Added impala_transactions.xml - Added impala_default_transactional_type.xml Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml A docs/topics/impala_default_transactional_type.xml A docs/topics/impala_transactions.xml 4 files changed, 670 insertions(+), 759 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/4 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14021 to look at the new patch set (#3). Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8813: [DOCS] Document the ACID support in Impala - Added impala_transactions.xml - Added impala_default_transactional_type.xml Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml A docs/topics/impala_default_transactional_type.xml A docs/topics/impala_transactions.xml 4 files changed, 670 insertions(+), 759 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/3 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/436/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 02:32:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8376: directory limits for scratch usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13986 ) Change subject: IMPALA-8376: directory limits for scratch usage .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Gerrit-Change-Number: 13986 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 01:56:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8376: directory limits for scratch usage
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13986 ) Change subject: IMPALA-8376: directory limits for scratch usage .. IMPALA-8376: directory limits for scratch usage This extends the --scratch_dirs syntax to support specifying a max capacity per directory, similarly to the --data_cache confirmation. The capacity is delimited from the directory name with ":" and uses the usual syntax for specifying memory. The following are valid arguments: * --scratch_dirs=/dir1,/dir2 (no limits) * --scratch_dirs=/dir1,/dir2:25G (only a limit on /dir2) * --scratch_dirs=/dir1:5MB,/dir2 (only a limit on /dir) * --scratch_dirs=/dir1:-1,/dir2:0 (alternative ways of expressing no limit) The usage is tracked with a metric per directory. Allocations from that directory start to fail when the limit is exceeded. These metrics are exposed as tmp-file-mgr.scratch-space-bytes-used.dir-0, tmp-file-mgr.scratch-space-bytes-used.dir-1, etc. Also add support for parsing terabyte specifiers to a utility function that is used for parsing many configurations. Testing: Added a unit test to exercise TmpFileMgr. Manually ran a spilling query on an impalad with multiple scratch dirs configured with different limits. Confirmed via metrics that the capacities were enforced. Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Reviewed-on: http://gerrit.cloudera.org:8080/13986 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/service/query-options-test.cc M be/src/util/parse-util-test.cc M be/src/util/parse-util.cc M common/thrift/generate_error_codes.py M common/thrift/metrics.json M docs/topics/impala_mem_limit.xml 10 files changed, 382 insertions(+), 41 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Gerrit-Change-Number: 13986 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4166/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 07 Aug 2019 01:46:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 01:44:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4165/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 07 Aug 2019 01:42:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 15 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 07 Aug 2019 01:41:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. IMPALA-8627: Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Test fixes: 1. Modified test_observability which fails on catalog-v2 since the profile emits different metadata load events. The test now looks for the right events on the profile depending on whether catalogv2 is enabled or not. 2. TableName.java constructor allows non-lowercased table and database names. This causes problems at the local catalog cache which expects the tablenames to be always in lowercase. More details on this failure are available in IMPALA-8627. The patch makes sure that the loadTable requests in local catalog do a explicit conversion of tablename to lowercase in order to get around the issue. 3. Fixes the JdbcTest which checks for existence of table comment in the getTables metadata jdbc call. In catalog-v2 since the columns are not requested, LocalTable is not loaded and hence the test needs to be modified to check if catalog-v2 is enabled. 4. Skips test_sanity which creates a Hive db and issues a invalidate metadata to make it visible in catalog. Unfortunately, in catalog-v2 currently there is no way to see a newly created database when event polling is disabled. 5. Similar to above (4) test_metadata_query_statements.py creates a hive db and issues a invalidate metadata. The test runs QueryTest/describe-db which is split into two one for checking the hive-db and other contains rest of the queries of the original describe-db. The split makes it possible to only execute the test partially when catalog-v2 is enabled Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Reviewed-on: http://gerrit.cloudera.org:8080/13933 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/workloads/functional-query/queries/QueryTest/describe-db.test A testdata/workloads/functional-query/queries/QueryTest/describe-hive-db.test M tests/common/environ.py M tests/common/skip.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_observability.py 16 files changed, 208 insertions(+), 68 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 16 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 2: I'm still thinking through the testing, but I thought I'd put this up. -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 07 Aug 2019 01:04:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4747/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 07 Aug 2019 01:04:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Joe McDonnell has posted comments on this change. (
http://gerrit.cloudera.org:8080/14026 )
Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small
clusters
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14026/1/be/src/scheduling/scheduler-test.cc
File be/src/scheduling/scheduler-test.cc:
http://gerrit.cloudera.org:8080/#/c/14026/1/be/src/scheduling/scheduler-test.cc@234
PS1, Line 234: for (int num_candidates = 1; num_candidates <=
num_impala_nodes + 2; ++num_candidates) {
> line too long (92 > 90)
Done
--
To view, visit http://gerrit.cloudera.org:8080/14026
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741
Gerrit-Change-Number: 14026
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Comment-Date: Wed, 07 Aug 2019 01:03:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14026 to look at the new patch set (#2). Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters In the original change for consistent scheduling, if a cluster has fewer nodes than the number of remote executor candidates, then the scheduler falls back to using the old SelectRemoteExecutor(). SelectRemoteExecutor() considers all backends and picks the backend with the least assigned bytes; to break ties, it uses randomness. This means that clusters with fewer backends than num_remote_executor_candidates do not have consistent placement. For the file handle cache (the original user of consistent placement), this is not a major problem. However, for data caching, it can result in slower warm up of the data cache and greater duplication of the same data across different nodes. This changes the algorithm to use consistent placement even for small clusters (num nodes <= num_remote_executor_candidates). This also changes GetRemoteExecutorCandidates() to return the candidates in the order that they were selected. In code, this means looping over a vector to detect distinct backends rather than using a set. Testing: - Modify the scheduler-test backend test to verify that small clusters use consistent scheduling. Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 --- M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc 2 files changed, 55 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/14026/2 -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14026 Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters In the original change for consistent scheduling, if a cluster has fewer nodes than the number of remote executor candidates, then the scheduler falls back to using the old SelectRemoteExecutor(). SelectRemoteExecutor() considers all backends and picks the backend with the least assigned bytes; to break ties, it uses randomness. This means that clusters with fewer backends than num_remote_executor_candidates do not have consistent placement. For the file handle cache (the original user of consistent placement), this is not a major problem. However, for data caching, it can result in slower warm up of the data cache and greater duplication of the same data across different nodes. This changes the algorithm to use consistent placement even for small clusters (num nodes <= num_remote_executor_candidates). This also changes GetRemoteExecutorCandidates() to return the candidates in the order that they were selected. In code, this means looping over a vector to detect distinct backends rather than using a set. Testing: - Modify the scheduler-test backend test to verify that small clusters use consistent scheduling. Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 --- M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc 2 files changed, 54 insertions(+), 43 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/14026/1 -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14026 )
Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small
clusters
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14026/1/be/src/scheduling/scheduler-test.cc
File be/src/scheduling/scheduler-test.cc:
http://gerrit.cloudera.org:8080/#/c/14026/1/be/src/scheduling/scheduler-test.cc@234
PS1, Line 234: for (int num_candidates = 1; num_candidates <=
num_impala_nodes + 2; ++num_candidates) {
line too long (92 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14026
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741
Gerrit-Change-Number: 14026
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Wed, 07 Aug 2019 01:01:41 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4742/ -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 07 Aug 2019 00:31:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4746/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 23:57:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 23:57:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 ) Change subject: IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4164/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14024 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd Gerrit-Change-Number: 14024 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 23:54:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14024 Change subject: IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus .. IMPALA-8766: Exorcise hadoop-cloud-storage + HWX Nexus Previous commits for IMPALA-8766 attempted to use hadoop-cloud-storage to satisfy Impala's cloud dependencies (e.g. hadoop-aws, hadoop-azure, etc). On builds with USE_CDP_HIVE=true, this adds Knox gateway-cloud-bindings. However, the entry for hadoop-cloud-storage artifact in the impala.cdp.repo maven repository introduces dependencies that are external to that repository. This requires the HWX Nexus repository to resolve those dangling dependencies. Unfortunately, HWX Nexus ages out old jars, including the ones we need. This stops using hadoop-cloud-storage, and instead adds a direct dependency to Knox for USE_CDP_HIVE=true. It disables the HWX Nexus repository and leaves a tombstone explaining why. Testing: - Deleted my .m2 directory and rebuilt Impala with USE_CDP_HIVE=true - Verified the CLASSPATH still contains the right jars on USE_CDP_HIVE=true Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd --- M bin/impala-config.sh M fe/pom.xml M impala-parent/pom.xml 3 files changed, 49 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/14024/1 -- To view, visit http://gerrit.cloudera.org:8080/14024 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd Gerrit-Change-Number: 14024 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4163/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 22:13:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14022 ) Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 21:56:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8781: Fix TestResultSpooling::test multi batches
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14022 Change subject: IMPALA-8781: Fix TestResultSpooling::test_multi_batches .. IMPALA-8781: Fix TestResultSpooling::test_multi_batches Prefix the query in TestResultSpooling::test_multi_batches with the database name. This was causing the Dockerized tests to fail. I double checked what other tests do and all the ones I saw either switch to the appropriate database or prefix the table name using the database name. The latter seemed more straightforward. I was not able to re-produce this locally, and its odd that this only affected the Dockerized tests (even more odd is that it seems to either be intermittent, or only affecting Dockerized tests triggered by gerrit-verify-dryrun-external). Regardless, it is a straightforward fix that makes the TestResultSpooling::test_multi_batches consistent with the rest of the tests. Testing: * Ran test_result_spooling.py locally using both bin/impala-py.test and tests/run-tests.py. Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c --- M tests/query_test/test_result_spooling.py 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/14022/1 -- To view, visit http://gerrit.cloudera.org:8080/14022 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I939eedba37003f5c720cea96e5c3532e2cc6312c Gerrit-Change-Number: 14022 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 21:34:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. IMPALA-8816: reduce custom cluster test runtime in core This includes some optimisations and a bulk move of tests to exhaustive. Move a bunch of custom cluster tests to exhaustive. I selected these partially based on runtime (i.e. I looked most carefully at the tests that ran for over a minute) and the likelihood of them catching a precommit bug. Regression tests for specific edge cases and tests for parts of the code that are very stable were prime candidates. Remove an unnecessary cluster restart in test_breakpad. Merge test_scheduler_error into test_failpoints to avoid an unnecessary cluster restart. Speed up cluster starts by ensuring that the default statestore args are applied even when _start_impala_cluster() is called directly. This shaves a couple of seconds off each restart. We made the default args use a faster update frequency - see IMPALA-7185 - but they did not take effect in all tests. Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Reviewed-on: http://gerrit.cloudera.org:8080/13967 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M tests/common/custom_cluster_test_suite.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_always_false_filter.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_breakpad.py M tests/custom_cluster/test_compact_catalog_updates.py M tests/custom_cluster/test_exchange_deferred_batches.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_query_event_hooks.py M tests/custom_cluster/test_redaction.py M tests/custom_cluster/test_scratch_disk.py M tests/custom_cluster/test_udf_concurrency.py M tests/custom_cluster/test_web_pages.py M tests/failure/test_failpoints.py 19 files changed, 140 insertions(+), 21 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8376: directory limits for scratch usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13986 ) Change subject: IMPALA-8376: directory limits for scratch usage .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4745/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Gerrit-Change-Number: 13986 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 21:18:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8376: directory limits for scratch usage
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13986 ) Change subject: IMPALA-8376: directory limits for scratch usage .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Gerrit-Change-Number: 13986 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 21:18:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8376: directory limits for scratch usage
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/13986 ) Change subject: IMPALA-8376: directory limits for scratch usage .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13986/3/be/src/runtime/tmp-file-mgr-test.cc File be/src/runtime/tmp-file-mgr-test.cc: http://gerrit.cloudera.org:8080/#/c/13986/3/be/src/runtime/tmp-file-mgr-test.cc@834 PS3, Line 834: EXPECT_EQ(4, dirs3.size()); > That is true, there isn't really anything smart to prevent that. Yea, I dont think we need to do anything clever here since the stakes are not high here. was just curious -- To view, visit http://gerrit.cloudera.org:8080/13986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I696146a65dbb97f1ba200ae472358ae2db6eb441 Gerrit-Change-Number: 13986 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 21:16:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: > I would prefer to keep this as a minimal change (version bump + > quick fixes for the tests it broke). I am ok with changing the > tests to set capabilities after that. Ok by me. May be create a separate JIRA to fix it for the python client -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 21:14:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 21:14:10 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: I would prefer to keep this as a minimal change (version bump + quick fixes for the tests it broke). I am ok with changing the tests to set capabilities after that. -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 20:57:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4162/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 14 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Tue, 06 Aug 2019 20:53:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4744/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 15 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 20:50:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7375: [DOCS] Added DATE functions
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13996 ) Change subject: IMPALA-7375: [DOCS] Added DATE functions .. Patch Set 2: (8 comments) Sorry for the very sloppy patch! http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@46 PS2, Line 46: is > are Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@78 PS2, Line 78: data > date Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@1072 PS2, Line 1072: MILLISECONDS > MILLISECOND Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@1180 PS2, Line 1180: '2019-08-17 > closing apostrophe is missing. Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@2159 PS2, Line 2159: arguemtn > argument Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@2249 PS2, Line 2249: IGINYTdays > BIGINT days Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@2939 PS2, Line 2939: 20019 > 2019 Done http://gerrit.cloudera.org:8080/#/c/13996/2/docs/topics/impala_datetime_functions.xml@3255 PS2, Line 3255:If date is not valid, e.g. DATE'2019-03-32', : returns NULL. > Actually, if the DATE literal is invalid, you get an error (here and elsewh Done -- To view, visit http://gerrit.cloudera.org:8080/13996 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1463731f9ee7fb9ec80d6be2458cffe5c42464b6 Gerrit-Change-Number: 13996 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Aug 2019 20:51:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 15 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 20:50:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4743/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 20:48:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 14: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/13933/11/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13933/11/tests/query_test/test_observability.py@340 PS11, Line 340: self.__verify_profile_event_sequence(load_event_regexes, runtime_profile) > Yeah, currently, they are in this order. We can change it to verify it they OK by me -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 20:41:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Tim Armstrong has abandoned this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13983/3/docs/topics/impala_date.xml File docs/topics/impala_date.xml: http://gerrit.cloudera.org:8080/#/c/13983/3/docs/topics/impala_date.xml@79 PS3, Line 79: Hive 3.1 and lower versions > Hive 3.1 doesn't have this issue (as correctly stated in L84), only version Done -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 20:20:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4161/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 20:19:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/435/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 20:16:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 1: Verified-1 Build Failed https://jenkins.impala.io/job/gerrit-docs-auto-test/434/ : Doc tests failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 20:15:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Jiawei Wang has posted comments on this change. (
http://gerrit.cloudera.org:8080/13801 )
Change subject: IMPALA-5149: Provide query profile in JSON format
..
Patch Set 14:
(14 comments)
> Uploaded patch set 14.
This has been a long process, thanks for your patience on this!
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/service/impala-server.cc@848
PS13, Line 848: Of
> offset: $1
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/service/impala-server.cc@2026
PS13, Line 2026: json_profile(rapidj
> json_profile. No need for "_rapid".
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile-test.cc
File be/src/util/runtime-profile-test.cc:
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile-test.cc@1166
PS13, Line 1166: RuntimeProfile* profile_ab = RuntimeProfile::Create(,
"ProfileAb");
> The name of the profile should probably match the variable name.
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile-test.cc@1204
PS13, Line 1204: for (auto& itr : content["counters"].GetArray()) {
: // check normal Counter
> Use iterator style like the following. Same at other places.
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile-test.cc@1218
PS13, Line 1218: }
> Do we want to check there are no other counter types in content ?
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile.h
File be/src/util/runtime-profile.h:
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile.h@622
PS13, Line 622: A map of counters name to counter
> A map of counter's name to counter
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile.h@625
PS13, Line 625: ame, const Counte
> This needs to be documented too.
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile.cc
File be/src/util/runtime-profile.cc:
http://gerrit.cloudera.org:8080/#/c/13801/13/be/src/util/runtime-profile.cc@704
PS13, Line 704: e information
> What does all the other information mean ?
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py
File tests/webserver/test_web_pages.py:
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py@571
PS13, Line 571:download_link =
"query_profile_plain_text?query_id={0}".format(query_id)
: assert down
> nit: one line
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py@587
PS13, Line 587:
> nit: indent 4 for line continuation
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py@594
PS13, Line 594: le:{0}".
> Downloaded ?
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py@595
PS13, Line 595: the query id
> Json pofile
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/tests/webserver/test_web_pages.py@597
PS13, Line 597:
> assert query_id in json_res["contents"]["profile_name"], json_res to help d
Done
http://gerrit.cloudera.org:8080/#/c/13801/13/www/query_profile.tmpl
File www/query_profile.tmpl:
http://gerrit.cloudera.org:8080/#/c/13801/13/www/query_profile.tmpl@28
PS13, Line 28: (Available Form
> (available formats):
Done
--
To view, visit http://gerrit.cloudera.org:8080/13801
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1
Gerrit-Change-Number: 13801
Gerrit-PatchSet: 14
Gerrit-Owner: Jiawei Wang
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: David Rorke
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jiawei Wang
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Tue, 06 Aug 2019 20:12:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Jiawei Wang has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. IMPALA-5149: Provide query profile in JSON format Description: Today there is a text and Thrift version of the query profile, but it would be useful to have a JSON version for portability and machine consumption. The ultimate goal is to have a Download JSON format profile link along with the other two formats. Modification: 1.Add Json format download option in impalad profile page 2.Add ToJson() function for RuntimeProfile, Counters, EventSequence 3.Add JSON format into QueryStateRecord 4.Add tests for E2E test, unit test, hs2 test 5.Modify query profile page to different download option 6.Modify HS2 server to support get JSON format profile Future compatibility: The schema of the JSON format can be changed in the future with the standardization of Profile and Counter structure. Tests: E2E tests: tests/webserver/test_web_pages.py - test_download_profile * merge text and json format download together HS2 tests: tests/hs2/test_hs2.py - test_get_profile * add json format test BE Unit tests: be/src/util/runtime-profile-test.cc * ToJson.RuntimeProfileToJsonTest * ToJson.EmptyTest * ToJson.EventSequenceToJsonTest * ToJson.TimeSeriesCounterToJsonTest Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/CMakeLists.txt M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/ImpalaService.thrift M common/thrift/RuntimeProfile.thrift M tests/hs2/test_hs2.py M tests/webserver/test_web_pages.py M www/query_profile.tmpl 16 files changed, 646 insertions(+), 60 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13801/14 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 14 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4160/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 13 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:59:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14021 to look at the new patch set (#2). Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8813: [DOCS] Document the ACID support in Impala Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml A docs/topics/impala_transactions.xml 3 files changed, 58 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/2 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/435/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 19:58:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/434/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 19:55:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8813: [DOCS] Document the ACID support in Impala
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14021 Change subject: IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8813: [DOCS] Document the ACID support in Impala Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml 2 files changed, 10 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/1 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4159/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:48:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4739/ -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 19:43:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4742/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 5 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:39:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 14: (1 comment) http://gerrit.cloudera.org:8080/#/c/13933/14/tests/hs2/hs2_test_suite.py File tests/hs2/hs2_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13933/14/tests/hs2/hs2_test_suite.py@74 PS14, Line 74: flake8: E251 unexpected spaces around keyword / parameter equals -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:39:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 14: (4 comments) http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py File tests/hs2/hs2_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py@29 PS13, Line 29: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py@72 PS13, Line 72: # decorators without installing new modules. > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py File tests/metadata/test_metadata_query_statements.py: http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py@180 PS13, Line 180: > flake8: E501 line too long (92 > 90 characters) Done http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py@181 PS13, Line 181: > flake8: E501 line too long (91 > 90 characters) Done -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:38:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. IMPALA-8627: Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Test fixes: 1. Modified test_observability which fails on catalog-v2 since the profile emits different metadata load events. The test now looks for the right events on the profile depending on whether catalogv2 is enabled or not. 2. TableName.java constructor allows non-lowercased table and database names. This causes problems at the local catalog cache which expects the tablenames to be always in lowercase. More details on this failure are available in IMPALA-8627. The patch makes sure that the loadTable requests in local catalog do a explicit conversion of tablename to lowercase in order to get around the issue. 3. Fixes the JdbcTest which checks for existence of table comment in the getTables metadata jdbc call. In catalog-v2 since the columns are not requested, LocalTable is not loaded and hence the test needs to be modified to check if catalog-v2 is enabled. 4. Skips test_sanity which creates a Hive db and issues a invalidate metadata to make it visible in catalog. Unfortunately, in catalog-v2 currently there is no way to see a newly created database when event polling is disabled. 5. Similar to above (4) test_metadata_query_statements.py creates a hive db and issues a invalidate metadata. The test runs QueryTest/describe-db which is split into two one for checking the hive-db and other contains rest of the queries of the original describe-db. The split makes it possible to only execute the test partially when catalog-v2 is enabled Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/workloads/functional-query/queries/QueryTest/describe-db.test A testdata/workloads/functional-query/queries/QueryTest/describe-hive-db.test M tests/common/environ.py M tests/common/skip.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_observability.py 16 files changed, 208 insertions(+), 68 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/14 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 14 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 13: (5 comments) http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py File tests/hs2/hs2_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py@29 PS13, Line 29: def add_session_helper(self, protocol_version, conf_overlay, close_session, fn): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py@72 PS13, Line 72: def needs_session_cluster_properties(protocol_version= flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13933/13/tests/hs2/hs2_test_suite.py@72 PS13, Line 72: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py File tests/metadata/test_metadata_query_statements.py: http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py@180 PS13, Line 180: 0 flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13933/13/tests/metadata/test_metadata_query_statements.py@181 PS13, Line 181: t flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 13 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:18:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 13: > Uploaded patch set 13. reverted test_hms_integration.py partially to avoid conflict with IMPALA-8795 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 13 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:18:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. IMPALA-8627: Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Test fixes: 1. Modified test_observability which fails on catalog-v2 since the profile emits different metadata load events. The test now looks for the right events on the profile depending on whether catalogv2 is enabled or not. 2. TableName.java constructor allows non-lowercased table and database names. This causes problems at the local catalog cache which expects the tablenames to be always in lowercase. More details on this failure are available in IMPALA-8627. The patch makes sure that the loadTable requests in local catalog do a explicit conversion of tablename to lowercase in order to get around the issue. 3. Fixes the JdbcTest which checks for existence of table comment in the getTables metadata jdbc call. In catalog-v2 since the columns are not requested, LocalTable is not loaded and hence the test needs to be modified to check if catalog-v2 is enabled. 4. Skips test_sanity which creates a Hive db and issues a invalidate metadata to make it visible in catalog. Unfortunately, in catalog-v2 currently there is no way to see a newly created database when event polling is disabled. 5. Similar to above (4) test_metadata_query_statements.py creates a hive db and issues a invalidate metadata. The test runs QueryTest/describe-db which is split into two one for checking the hive-db and other contains rest of the queries of the original describe-db. The split makes it possible to only execute the test partially when catalog-v2 is enabled Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/workloads/functional-query/queries/QueryTest/describe-db.test A testdata/workloads/functional-query/queries/QueryTest/describe-hive-db.test M tests/common/environ.py M tests/common/skip.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_observability.py 16 files changed, 206 insertions(+), 68 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/13 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 13 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8761 Configuration validation introduced in IMPALA-8559 can be improved
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/13952 )
Change subject: IMPALA-8761 Configuration validation introduced in IMPALA-8559
can be improved
..
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java:
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@281
PS1, Line 281: results.size(
Instead of printing the ArrayList directly, can you add a log at error level
here directly LOG.error("Found {} metastore configuration(s) incorrectly,
results.size()) and then the loop below just do a
LOG.error(invalidConfig.getReason());
If you do that way, you can also get rid of List errorMessages.
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@286
PS1, Line 286: throw new CatalogException(String.format("Found %d
metastore configuration(s) incorrectly"+"" +
> line too long (101 > 90)
Please fix this line width issue. Use the IDE to reformat the changed code
block.
http://gerrit.cloudera.org:8080/#/c/13952/1/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java@287
PS1, Line 287: " set. Event processing cannot be started. See
error log for more details.", results.size()));
> line too long (108 > 90)
Please fix this line width issue. Use the IDE to reformat the changed code
block.
--
To view, visit http://gerrit.cloudera.org:8080/13952
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I52d07bce88f9332a34bfe2f9b31570203485d544
Gerrit-Change-Number: 13952
Gerrit-PatchSet: 1
Gerrit-Owner: Sharanitha Harish
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 06 Aug 2019 19:16:26 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 12: (5 comments) http://gerrit.cloudera.org:8080/#/c/13933/12/tests/hs2/hs2_test_suite.py File tests/hs2/hs2_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13933/12/tests/hs2/hs2_test_suite.py@29 PS12, Line 29: def add_session_helper(self, protocol_version, conf_overlay, close_session, fn): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13933/12/tests/hs2/hs2_test_suite.py@72 PS12, Line 72: def needs_session_cluster_properties(protocol_version= flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13933/12/tests/hs2/hs2_test_suite.py@72 PS12, Line 72: flake8: E251 unexpected spaces around keyword / parameter equals http://gerrit.cloudera.org:8080/#/c/13933/12/tests/metadata/test_metadata_query_statements.py File tests/metadata/test_metadata_query_statements.py: http://gerrit.cloudera.org:8080/#/c/13933/12/tests/metadata/test_metadata_query_statements.py@180 PS12, Line 180: 0 flake8: E501 line too long (92 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13933/12/tests/metadata/test_metadata_query_statements.py@181 PS12, Line 181: t flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:08:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has uploaded a new patch set (#12). ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. IMPALA-8627: Enable catalog-v2 in tests This patch enables catalog-v2 by default in all the tests. Test fixes: 1. Modified test_observability which fails on catalog-v2 since the profile emits different metadata load events. The test now looks for the right events on the profile depending on whether catalogv2 is enabled or not. 2. TableName.java constructor allows non-lowercased table and database names. This causes problems at the local catalog cache which expects the tablenames to be always in lowercase. More details on this failure are available in IMPALA-8627. The patch makes sure that the loadTable requests in local catalog do a explicit conversion of tablename to lowercase in order to get around the issue. 3. Fixes the JdbcTest which checks for existence of table comment in the getTables metadata jdbc call. In catalog-v2 since the columns are not requested, LocalTable is not loaded and hence the test needs to be modified to check if catalog-v2 is enabled. 4. Skips test_sanity which creates a Hive db and issues a invalidate metadata to make it visible in catalog. Unfortunately, in catalog-v2 currently there is no way to see a newly created database when event polling is disabled. 5. Similar to above (4) test_metadata_query_statements.py creates a hive db and issues a invalidate metadata. The test runs QueryTest/describe-db which is split into two one for checking the hive-db and other contains rest of the queries of the original describe-db. The split makes it possible to only execute the test partially when catalog-v2 is enabled Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d --- M docker/catalogd/Dockerfile M docker/impalad_coord_exec/Dockerfile M docker/impalad_coordinator/Dockerfile M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/workloads/functional-query/queries/QueryTest/describe-db.test A testdata/workloads/functional-query/queries/QueryTest/describe-hive-db.test M tests/common/environ.py M tests/common/skip.py M tests/hs2/hs2_test_suite.py M tests/hs2/test_hs2.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py M tests/query_test/test_observability.py 16 files changed, 209 insertions(+), 71 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/13933/12 -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 12 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8627: Enable catalog-v2 in tests
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13933 ) Change subject: IMPALA-8627: Enable catalog-v2 in tests .. Patch Set 11: (4 comments) Addressed Tim's comments and suggestions. http://gerrit.cloudera.org:8080/#/c/13933/11/tests/hs2/hs2_test_suite.py File tests/hs2/hs2_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13933/11/tests/hs2/hs2_test_suite.py@74 PS11, Line 74: def add_session(self, cluster_properties): > I think we can avoid all of the duplication with the above add session func Thanks for sharing your the code snippet. Incorporated into the patch. http://gerrit.cloudera.org:8080/#/c/13933/11/tests/hs2/test_hs2.py File tests/hs2/test_hs2.py: http://gerrit.cloudera.org:8080/#/c/13933/11/tests/hs2/test_hs2.py@440 PS11, Line 440: @needs_session_cluster_properties() > If we're defining a new decorator anyway, could we pass in unique_database Done http://gerrit.cloudera.org:8080/#/c/13933/11/tests/query_test/test_observability.py File tests/query_test/test_observability.py: http://gerrit.cloudera.org:8080/#/c/13933/11/tests/query_test/test_observability.py@318 PS11, Line 318: r'CatalogFetch.Config.Misses|CatalogFetch.Config.Hits', > Is it guaranteed that we always get one or the other? yes, according to https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java#L516 Its always either a hit or miss and hence one of the counter will be added. I ran this test multiple times which generates a miss event first and then later on hit event subsequently. http://gerrit.cloudera.org:8080/#/c/13933/11/tests/query_test/test_observability.py@340 PS11, Line 340: self.__verify_profile_event_sequence(load_event_regexes, runtime_profile) > Do we need to test that the above regexes are in sequential lines, or just Yeah, currently, they are in this order. We can change it to verify it they are present in different order by sort them out but it needed some more plumbing. I would like to defer it to later when its really needed. -- To view, visit http://gerrit.cloudera.org:8080/13933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iddbde666de2b780c0e40df716a9dfe54524e092d Gerrit-Change-Number: 13933 Gerrit-PatchSet: 11 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Tue, 06 Aug 2019 19:05:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4158/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 18:05:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4157/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:59:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: AnalyzerTest.TestAnalyzeTransactional() test fix
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14017 ) Change subject: IMPALA-8600: AnalyzerTest.TestAnalyzeTransactional() test fix .. IMPALA-8600: AnalyzerTest.TestAnalyzeTransactional() test fix Adjusts expected error message in AnalyzerTest.TestAnalyzeTransactional() after rewriting the message. Change-Id: I7f1ed5da8cd3511eae4db12fb5ce1235aee50fd6 Reviewed-on: http://gerrit.cloudera.org:8080/14017 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14017 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7f1ed5da8cd3511eae4db12fb5ce1235aee50fd6 Gerrit-Change-Number: 14017 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8600: AnalyzerTest.TestAnalyzeTransactional() test fix
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14017 ) Change subject: IMPALA-8600: AnalyzerTest.TestAnalyzeTransactional() test fix .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14017 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7f1ed5da8cd3511eae4db12fb5ce1235aee50fd6 Gerrit-Change-Number: 14017 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 17:48:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4740/ -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:34:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4741/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:34:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13967 to look at the new patch set (#9). Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. IMPALA-8816: reduce custom cluster test runtime in core This includes some optimisations and a bulk move of tests to exhaustive. Move a bunch of custom cluster tests to exhaustive. I selected these partially based on runtime (i.e. I looked most carefully at the tests that ran for over a minute) and the likelihood of them catching a precommit bug. Regression tests for specific edge cases and tests for parts of the code that are very stable were prime candidates. Remove an unnecessary cluster restart in test_breakpad. Merge test_scheduler_error into test_failpoints to avoid an unnecessary cluster restart. Speed up cluster starts by ensuring that the default statestore args are applied even when _start_impala_cluster() is called directly. This shaves a couple of seconds off each restart. We made the default args use a faster update frequency - see IMPALA-7185 - but they did not take effect in all tests. Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 --- M tests/common/custom_cluster_test_suite.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_always_false_filter.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_breakpad.py M tests/custom_cluster/test_compact_catalog_updates.py M tests/custom_cluster/test_exchange_deferred_batches.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_query_event_hooks.py M tests/custom_cluster/test_redaction.py M tests/custom_cluster/test_scratch_disk.py M tests/custom_cluster/test_udf_concurrency.py M tests/custom_cluster/test_web_pages.py M tests/failure/test_failpoints.py 19 files changed, 140 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/13967/9 -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:34:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/13967/8/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/13967/8/tests/common/impala_service.py@256 PS8, Line 256: cannot > can Done -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:33:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 8: (1 comment) One final nit http://gerrit.cloudera.org:8080/#/c/13967/8/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/13967/8/tests/common/impala_service.py@256 PS8, Line 256: cannot can -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:28:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4740/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:18:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13967 ) Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/13967/6//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13967/6//COMMIT_MSG@24 PS6, Line 24: Speed up cluster starts by ensuring that the default statestore args are : applied even when _start_impala_cluster() is called directly. This : shaves a couple of seconds off each restart. > Can you add a few words giving some context about why the default args spee Done http://gerrit.cloudera.org:8080/#/c/13967/6/tests/common/impala_service.py File tests/common/impala_service.py: http://gerrit.cloudera.org:8080/#/c/13967/6/tests/common/impala_service.py@256 PS6, Line 256: cannot : throw and exception > can throw an exception Done -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 6 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 17:18:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8816: reduce custom cluster test runtime in core
Hello Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13967 to look at the new patch set (#7). Change subject: IMPALA-8816: reduce custom cluster test runtime in core .. IMPALA-8816: reduce custom cluster test runtime in core This includes some optimisations and a bulk move of tests to exhaustive. Move a bunch of custom cluster tests to exhaustive. I selected these partially based on runtime (i.e. I looked most carefully at the tests that ran for over a minute) and the likelihood of them catching a precommit bug. Regression tests for specific edge cases and tests for parts of the code that are very stable were prime candidates. Remove an unnecessary cluster restart in test_breakpad. Merge test_scheduler_error into test_failpoints to avoid an unnecessary cluster restart. Speed up cluster starts by ensuring that the default statestore args are applied even when _start_impala_cluster() is called directly. This shaves a couple of seconds off each restart. We made the default args use a faster update frequency - see IMPALA-7185 - but they did not take effect in all tests. Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 --- M tests/common/custom_cluster_test_suite.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/custom_cluster/test_admission_controller.py M tests/custom_cluster/test_always_false_filter.py M tests/custom_cluster/test_auto_scaling.py M tests/custom_cluster/test_breakpad.py M tests/custom_cluster/test_compact_catalog_updates.py M tests/custom_cluster/test_exchange_deferred_batches.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/custom_cluster/test_metadata_replicas.py M tests/custom_cluster/test_parquet_max_page_header.py M tests/custom_cluster/test_permanent_udfs.py M tests/custom_cluster/test_query_event_hooks.py M tests/custom_cluster/test_redaction.py M tests/custom_cluster/test_scratch_disk.py M tests/custom_cluster/test_udf_concurrency.py M tests/custom_cluster/test_web_pages.py M tests/failure/test_failpoints.py 19 files changed, 140 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/67/13967/7 -- To view, visit http://gerrit.cloudera.org:8080/13967 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib2e3e7ebc9695baec4d69183387259958df10f62 Gerrit-Change-Number: 13967 Gerrit-PatchSet: 7 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: (1 comment) Looks good, just had one nit about the test http://gerrit.cloudera.org:8080/#/c/14019/4/be/src/util/dict-test.cc File be/src/util/dict-test.cc: http://gerrit.cloudera.org:8080/#/c/14019/4/be/src/util/dict-test.cc@287 PS4, Line 287: EXPECT_TRUE(status.ok()); We have an EXPECT_OK macro for this that will print the error if the check fails. -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 16:39:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8771: Missing stats warning for complex type columns
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13965 ) Change subject: IMPALA-8771: Missing stats warning for complex type columns .. Patch Set 8: Verified+1 Code-Review+2 Carry +2 and verified across rebase -- To view, visit http://gerrit.cloudera.org:8080/13965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia1b5c14da0c7f6eab373d80b2dbf7c974b2eb567 Gerrit-Change-Number: 13965 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 16:38:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8771: Missing stats warning for complex type columns
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13965 ) Change subject: IMPALA-8771: Missing stats warning for complex type columns .. IMPALA-8771: Missing stats warning for complex type columns An extra condition is added to the table stats checking, so that the complex type columns are skipped and can not trigger missing stats warning. Change-Id: Ia1b5c14da0c7f6eab373d80b2dbf7c974b2eb567 Reviewed-on: http://gerrit.cloudera.org:8080/13965 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M fe/src/main/java/org/apache/impala/planner/ScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test A testdata/workloads/functional-query/queries/QueryTest/compute-stats-complextype-warning.test M testdata/workloads/functional-query/queries/QueryTest/show-stats.test M tests/metadata/test_compute_stats.py 5 files changed, 32 insertions(+), 27 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia1b5c14da0c7f6eab373d80b2dbf7c974b2eb567 Gerrit-Change-Number: 13965 Gerrit-PatchSet: 9 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8771: Missing stats warning for complex type columns
Tim Armstrong has uploaded a new patch set (#8) to the change originally created by Tamas Mate. ( http://gerrit.cloudera.org:8080/13965 ) Change subject: IMPALA-8771: Missing stats warning for complex type columns .. IMPALA-8771: Missing stats warning for complex type columns An extra condition is added to the table stats checking, so that the complex type columns are skipped and can not trigger missing stats warning. Change-Id: Ia1b5c14da0c7f6eab373d80b2dbf7c974b2eb567 --- M fe/src/main/java/org/apache/impala/planner/ScanNode.java M testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test A testdata/workloads/functional-query/queries/QueryTest/compute-stats-complextype-warning.test M testdata/workloads/functional-query/queries/QueryTest/show-stats.test M tests/metadata/test_compute_stats.py 5 files changed, 32 insertions(+), 27 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13965/8 -- To view, visit http://gerrit.cloudera.org:8080/13965 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia1b5c14da0c7f6eab373d80b2dbf7c974b2eb567 Gerrit-Change-Number: 13965 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: > From yesterdays FE meeting, Vihang prefer following fix: > --- a/tests/common/impala_test_suite.py > +++ b/tests/common/impala_test_suite.py > @@ -81,6 +81,7 @@ from tests.util.thrift_util import > create_transport > > # Imports required for Hive Metastore Client > from hive_metastore import ThriftHiveMetastore > +from hive_metastore import ttypes > from thrift.protocol import TBinaryProtocol > > # Initializing the logger before conditional imports, since we will > need it > @@ -122,7 +123,6 @@ SET_PATTERN = re.compile( > METRICS_URL = 'http://localhost:25000/metrics?json' > > GROUP_NAME = grp.getgrgid(pwd.getpwnam(getuser()).pw_gid).gr_name > - > # Base class for Impala tests. All impala test cases should inherit > from this class > class ImpalaTestSuite(BaseTestSuite): > @classmethod > @@ -806,7 +806,14 @@ class ImpalaTestSuite(BaseTestSuite): > table = self.hive_client.get_table(db_name, table_name) > assert table is not None > self.hive_client.drop_table(db_name, table_name, True) > -self.hive_client.create_table(table) > +if HIVE_MAJOR_VERSION > 2: > + capabilities = ["EXTREAD", "EXTWRITE","HIVEBUCKET2", > "HIVEMANAGEDINSERTREAD", > + "HIVEMANAGEDINSERTWRITE", "HIVEMQT", "HIVEFULLACIDREAD", > "HIVEFULLACIDWRITE"] > + createTblRequest = ttypes.CreateTableRequest(table, > + processorCapabilities=capabilities, > processorIdentifier="ImpalatestPython") > + self.hive_client.create_table_req(createTblRequest) > +else: > + self.hive_client.create_table(table) Thanks Yonghzi .. The above code looks right to me. I would update the comment in the MetastoreShim setCapabilities() so that we are aware that any change in that method would like mean a change in this part of the code as well. Not sure if we have to do this for other APIs as well? I see in HiveMetastoreClient we send the capabilities in getPartitionByNames, and some getTable API variations too. We don't have to implement it in all the APIs but only the ones which we use. -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 16:36:43 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 3: From yesterdays FE meeting, Vihang prefer following fix: --- a/tests/common/impala_test_suite.py +++ b/tests/common/impala_test_suite.py @@ -81,6 +81,7 @@ from tests.util.thrift_util import create_transport # Imports required for Hive Metastore Client from hive_metastore import ThriftHiveMetastore +from hive_metastore import ttypes from thrift.protocol import TBinaryProtocol # Initializing the logger before conditional imports, since we will need it @@ -122,7 +123,6 @@ SET_PATTERN = re.compile( METRICS_URL = 'http://localhost:25000/metrics?json' GROUP_NAME = grp.getgrgid(pwd.getpwnam(getuser()).pw_gid).gr_name - # Base class for Impala tests. All impala test cases should inherit from this class class ImpalaTestSuite(BaseTestSuite): @classmethod @@ -806,7 +806,14 @@ class ImpalaTestSuite(BaseTestSuite): table = self.hive_client.get_table(db_name, table_name) assert table is not None self.hive_client.drop_table(db_name, table_name, True) -self.hive_client.create_table(table) +if HIVE_MAJOR_VERSION > 2: + capabilities = ["EXTREAD", "EXTWRITE","HIVEBUCKET2", "HIVEMANAGEDINSERTREAD", + "HIVEMANAGEDINSERTWRITE", "HIVEMQT", "HIVEFULLACIDREAD", "HIVEFULLACIDWRITE"] + createTblRequest = ttypes.CreateTableRequest(table, + processorCapabilities=capabilities, processorIdentifier="ImpalatestPython") + self.hive_client.create_table_req(createTblRequest) +else: + self.hive_client.create_table(table) -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Tue, 06 Aug 2019 16:29:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4156/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 06 Aug 2019 16:27:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/13968/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java: http://gerrit.cloudera.org:8080/#/c/13968/5/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@566 PS5, Line 566: Thread.sleep(retryWaitSeconds * 1000); TODO in next PS: put this Thread.sleep() before checkLock() -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Aug 2019 16:09:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4155/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Aug 2019 15:50:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14004 Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. IMPALA-8796: Add unit tests to UnpackAndDecodeValues Adding unit tests to test BitPacking::UnpackAndDecodeValues. Also changing the existing BitPacking tests so that they are parametrised by the output type of the unpacking so we can test different output types. Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 --- M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.cc 2 files changed, 190 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14004/2 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4154/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Tue, 06 Aug 2019 15:40:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#5). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. TODOs: * add tests * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java 8 files changed, 465 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/5 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 5 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4152/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 15:09:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4153/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 15:05:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13968 )
Change subject: IMPALA-8637: Implement transaction handling and locking for
ACID queries
..
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/13968/3/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/13968/3/be/src/service/client-request-state.cc@1129
PS3, Line 1129: int64_t txn_id = GetTransactionId();
> Maybe log the error? Should be pretty rare and might be useful on the off c
Done
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@522
PS3, Line 522: client.heartbeat(txnId, lockId);
> We get the query id as part of the log message, right?
Currently not. Do you know how we do that in the Frontend, e.g.? I'm not sure
if we could do the same trick since we are on a different thread here that
heartbeats multiple in behalf of multiple queries.
I added a HeartbeatContext object for each transaction and lock in the map, and
I explicitly added the query_id to the log messages in TransactionKeepalive.
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@553
PS3, Line 553: lockRequestBuilder.setTransactionId(txnId);
: for (LockComponent lockComponent :
> lock() may also return state WAITING according to https://github.com/apache
Thanks for catching this. Added some retrying and waiting logic to this
function.
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/main/java/org/apache/impala/service/Frontend.java@265
PS3, Line 265:
> Is there a reason this liven in Frontend class? The reason I ask is that I
We talked about this offline. So the original concept was that the Coordinator
is responsible for managing transactions, locks, and heartbeats.
But maybe for some statements it'd make sense to execute them on the Catalog
side, e.g. DROP and TRUNCATE. Basically we could just put the
TransactionKeepalive class to common and have both the Frontend and
CatalogOpExecutor could have an instance of it.
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/main/java/org/apache/impala/service/Frontend.java@1743
PS3, Line 1743: ().getName(
> Shouldn't this be SHARED_WRITE?
No, SHARED_WRITE is for UPDATE and DELETE.
https://github.com/apache/hive/blob/959ebeb680b07d59f4f55939862ebbc2d7f16a92/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#L2873
http://gerrit.cloudera.org:8080/#/c/13968/2/fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java
File fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java:
http://gerrit.cloudera.org:8080/#/c/13968/2/fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java@73
PS2, Line 73: private class DaemonThread implements Runnable {
> Yeah I think adding some jitter would be useful, e.g. if you sleep for a ra
Added some jitter as suggested.
Yeah actually I'm familiar with the session expiry logic and initially I was
thinking about implementing stg similar. But I wanted to start with a simple
solution. Also, currently we only open transactions for INSERTs.
Added a TODO to implement a smarter solution when we start putting queries into
transactions.
Btw, even when we put every query into a transaction it doesn't necessarily
mean that we open a transaction for each one of them, i.e. a bunch of queries
could just re-use the same transaction id. That'd complicate locking a bit
though.
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java
File fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java:
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java@93
PS3, Line 93: pyOf
> I think that this should be error as the program really shouldn't get here.
Done
--
To view, visit http://gerrit.cloudera.org:8080/13968
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa
Gerrit-Change-Number: 13968
Gerrit-PatchSet: 4
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Tue, 06 Aug 2019 15:00:07
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#4). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. TODOs: * add tests * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java 8 files changed, 464 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/4 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4739/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 06 Aug 2019 14:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch()
Daniel Becker has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/14019 ) Change subject: IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() .. IMPALA-8833: Check failed in BatchedBitReader::UnpackBatch() After raising the maximum bit width for bit packing to 64 bits, DictDecoder accepted bit widths between 32 and 64, but internally it uses 32 bit integers and unpacking ran into a DCHECK. Adding a check to DictDecoder to catch if the bit width is higher than 32. Testing: Added a test that asserts that DictDecoder accepts bit widths 0-32 and rejects higher bit widths which could still be unpacked otherwise. Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 --- M be/src/util/dict-encoding.h M be/src/util/dict-test.cc 2 files changed, 31 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/14019/4 -- To view, visit http://gerrit.cloudera.org:8080/14019 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4cba3338a93f8287c24abbe3ad9bfcbfa756bca4 Gerrit-Change-Number: 14019 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
