[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 7: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4756/ -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 08 Aug 2019 02:14:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 11 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 08 Aug 2019 01:05:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. IMPALA-8549: Add support for scanning DEFLATE text files This patch adds support to Impala for scanning .DEFLATE files of tables stored as text. To avoid confusion, it should be noted that although these files have a compression type of DEFLATE in Impala, they should be treated as if their compression type is DEFAULT. Hadoop tools such as Hive and MapReduce support reading and writing text files compressed using the deflate algorithm, which is the default compression type. Hadoop uses the zlib library (an implementation of the DEFLATE algorithm) to compress text files into .DEFLATE files, which are not in the raw deflate format but rather the zlib format (the zlib library supports three flavors of deflate, and Hadoop uses the flavor that compresses data into deflate with zlib wrappings rather than just raw deflate) Testing: There is a pre-existing unit test that validates compressing and decompressing data with compression type DEFLATE. Also, modified existing end-to-end testing that simulates querying files of various formats and compression types. All core and exhaustive tests pass. Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Reviewed-on: http://gerrit.cloudera.org:8080/13857 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M be/src/exec/hdfs-text-scanner.cc M be/src/exec/hdfs-text-scanner.h M testdata/datasets/functional/schema_constraints.csv M testdata/workloads/functional-query/functional-query_exhaustive.csv M tests/query_test/test_compressed_formats.py 5 files changed, 25 insertions(+), 19 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 12 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 11 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 08 Aug 2019 00:26:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13992 ) Change subject: IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator .. Patch Set 2: (4 comments) http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/admission-controller.cc@442 PS2, Line 442: && CanMemLimitAccommodateReservation( Nice to remove this special case. http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/query-schedule.h File be/src/scheduling/query-schedule.h: http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/query-schedule.h@81 PS2, Line 81: // Indicates whether this backend will run on the coordinator backend. nit: wording is a bit weird. Maybe just "Indicates whether this backend is the coordinator". http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/query-schedule.h@374 PS2, Line 374: requiresCoordinatorFragment nit: RequiresCoordinatorFragment() http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/query-schedule.cc File be/src/scheduling/query-schedule.cc: http://gerrit.cloudera.org:8080/#/c/13992/2/be/src/scheduling/query-schedule.cc@261 PS2, Line 261: only_coord_fragment_schduled scheduled? -- To view, visit http://gerrit.cloudera.org:8080/13992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2 Gerrit-Change-Number: 13992 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 23:45:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14034 ) Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/441/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 23:01:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14034 ) Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/440/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 23:00:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4179/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 07 Aug 2019 22:47:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14034 ) Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/441/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 22:42:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13968 )
Change subject: IMPALA-8637: Implement transaction handling and locking for
ACID queries
..
Patch Set 10: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13968/3/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@522
PS3, Line 522: client.heartbeat(txnId, lockId);
> Currently not. Do you know how we do that in the Frontend, e.g.? I'm not su
Ah yeah, you're totally right. Thanks for fixing.
http://gerrit.cloudera.org:8080/#/c/13968/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:
http://gerrit.cloudera.org:8080/#/c/13968/10/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@567
PS10, Line 567: } catch (InterruptedException e) {
Leave a comment here to explain why it isn't a bug (empty catch blocks
automatically look like a bug to me)
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py@116
PS10, Line 116: if
self._any_open_heartbeated_transaction_since(latest_open_txn):
Nice!
--
To view, visit http://gerrit.cloudera.org:8080/13968
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa
Gerrit-Change-Number: 13968
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 07 Aug 2019 22:42:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Hello Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14034 to look at the new patch set (#2). Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b --- M docs/topics/impala_shell_options.xml 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14034/2 -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14034 Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b --- M docs/topics/impala_shell_options.xml 1 file changed, 5 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/14034/1 -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14034 ) Change subject: IMPALA-8334: [DOCS] Document the keyval option for impala-shell config file .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/440/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14034 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1067dec375c38eed857834d2cf24cb2b86b Gerrit-Change-Number: 14034 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 22:37:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 7: (24 comments) http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/environ.py File tests/common/environ.py: http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/environ.py@332 PS7, Line 332: w flake8: E501 line too long (101 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/environ.py@341 PS7, Line 341: , flake8: E501 line too long (117 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/environ.py@350 PS7, Line 350: i flake8: F821 undefined name 'impala_url' http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/environ.py@384 PS7, Line 384: r flake8: E501 line too long (102 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/impala_test_suite.py@971 PS7, Line 971: l flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/common/impala_test_suite.py@971 PS7, Line 971: """Confirm the database with 'db_name' is present in the impalad's local catalog. line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/custom_cluster/test_hive_parquet_codec_interop.py File tests/custom_cluster/test_hive_parquet_codec_interop.py: http://gerrit.cloudera.org:8080/#/c/13922/7/tests/custom_cluster/test_hive_parquet_codec_interop.py@91 PS7, Line 91: E flake8: F821 undefined name 'EventProcessorUtils' http://gerrit.cloudera.org:8080/#/c/13922/7/tests/metadata/test_refresh_partition.py File tests/metadata/test_refresh_partition.py: http://gerrit.cloudera.org:8080/#/c/13922/7/tests/metadata/test_refresh_partition.py@66 PS7, Line 66: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/metadata/test_refresh_partition.py@66 PS7, Line 66: EventProcessorUtils.wait_for_event_processing(self.hive_client) line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py File tests/util/event_processor_utils.py: http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@22 PS7, Line 22: import logging flake8: F401 'logging' imported but unused http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@28 PS7, Line 28: class EventProcessorUtils(object): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@31 PS7, Line 31: @ flake8: E301 expected 1 blank line, found 0 http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@47 PS7, Line 47: T flake8: F821 undefined name 'TimeoutError' http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@49 PS7, Line 49: . flake8: E131 continuation line unaligned for hanging indent http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@53 PS7, Line 53: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@53 PS7, Line 53: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@54 PS7, Line 54: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@54 PS7, Line 54: @staticmethod line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@62 PS7, Line 62: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@62 PS7, Line 62: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@66 PS7, Line 66: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@66 PS7, Line 66: line has trailing whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@69 PS7, Line 69: flake8: W293 blank line contains whitespace http://gerrit.cloudera.org:8080/#/c/13922/7/tests/util/event_processor_utils.py@69 PS7, Line 69: line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 07 Aug 2019 22:07:12 + Gerrit-HasComments:
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4756/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 07 Aug 2019 22:06:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8795 : Enable event polling by default in tests
Vihang Karajgaonkar has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/13922 ) Change subject: IMPALA-8795 : Enable event polling by default in tests .. IMPALA-8795 : Enable event polling by default in tests Most of the fixes for event processing are committed. This change enables the feature by default for all the tests so that eventually we can turn it on out of the box. Testing done [WIP]: 1. Ran all the tests with the patch and confirmed that there are no test failures 2. Ran all the tests with CDP build. Change-Id: I7279349d4900e24fbcf558f290549496844ce138 --- M docker/catalogd/Dockerfile M tests/common/environ.py M tests/common/impala_test_suite.py M tests/custom_cluster/test_hive_parquet_codec_interop.py M tests/metadata/test_hms_integration.py M tests/metadata/test_metadata_query_statements.py M tests/metadata/test_refresh_partition.py A tests/util/event_processor_utils.py 8 files changed, 157 insertions(+), 37 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/13922/7 -- To view, visit http://gerrit.cloudera.org:8080/13922 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7279349d4900e24fbcf558f290549496844ce138 Gerrit-Change-Number: 13922 Gerrit-PatchSet: 7 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13992 ) Change subject: IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2 Gerrit-Change-Number: 13992 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 21:22:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 15: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 15 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 07 Aug 2019 21:10:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. IMPALA-5149: Provide query profile in JSON format Description: Today there is a text and Thrift version of the query profile, but it would be useful to have a JSON version for portability and machine consumption. The ultimate goal is to have a Download JSON format profile link along with the other two formats. Modification: 1.Add Json format download option in impalad profile page 2.Add ToJson() function for RuntimeProfile, Counters, EventSequence 3.Add JSON format into QueryStateRecord 4.Add tests for E2E test, unit test, hs2 test 5.Modify query profile page to different download option 6.Modify HS2 server to support get JSON format profile Future compatibility: The schema of the JSON format can be changed in the future with the standardization of Profile and Counter structure. Tests: E2E tests: tests/webserver/test_web_pages.py - test_download_profile * merge text and json format download together HS2 tests: tests/hs2/test_hs2.py - test_get_profile * add json format test BE Unit tests: be/src/util/runtime-profile-test.cc * ToJson.RuntimeProfileToJsonTest * ToJson.EmptyTest * ToJson.EventSequenceToJsonTest * ToJson.TimeSeriesCounterToJsonTest Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Reviewed-on: http://gerrit.cloudera.org:8080/13801 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/CMakeLists.txt M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile-test.cc M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/ImpalaService.thrift M common/thrift/RuntimeProfile.thrift M tests/hs2/test_hs2.py M tests/webserver/test_web_pages.py M www/query_profile.tmpl 16 files changed, 646 insertions(+), 60 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 16 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13992 Change subject: IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator .. IMPALA-8791: Handle the case where there is no fragment scheduled on the coordinator This patch fixes a bug where if an insert or CTAS query has no fragments scheduled on the coordinator and a mem limit is to be enforced on the query (either through query option or automatically through estimates) then the same limit is also applied to the coordinator backend even though it does not execute anything. Highlights: - coord_backend_mem_to_admit_/mem_limit will always refer to the memory to admit/limit for the coordinator regardless of which fragments are scheduled on it. - There will always be a BackendExecParams added for the coordinator because coordinator always spawns a QueryState object with a mem_tracker for tracking runtime filter mem and the result set cache. For the case where this BackendExecParams is empty (no instances scheduled) it would ensure that some minimal amount of memory is accounted for by the admission controller and the right mem limit is applied to the QueryState spawned by the coordinator - added changes to Coordinator and Coordinator::BackendState classes to handle an empty BackendExecParams object Testing: The following cases need to be tested where the kind of fragments schduled on the coordinator backend are: 1. Coordinator fragment + other exec fragments 2. Coordinator fragment only 3. other exec fragments only (eg. insert into values OR insert into select 1) 4. No fragments, but coordinator still creates a QueryState Case 1 is covered by tests working with non-dedicated coordinators. Rest are covered by test_mem_limit_dedicated_coordinator in test_admission_controller.py Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/query-schedule.cc M be/src/scheduling/query-schedule.h M be/src/scheduling/scheduler.cc M be/src/scheduling/scheduler.h M fe/src/main/java/org/apache/impala/planner/Planner.java M testdata/workloads/functional-query/queries/QueryTest/dedicated-coord-mem-estimates.test 12 files changed, 151 insertions(+), 120 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/13992/2 -- To view, visit http://gerrit.cloudera.org:8080/13992 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If5631fa1490d9612ffac3c4c4715348de47d6df2 Gerrit-Change-Number: 13992 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14033 ) Change subject: IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/439/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I591e0fd736ea114aa52a999acf41806a94e49382 Gerrit-Change-Number: 14033 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 07 Aug 2019 20:33:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8549: Add support for scanning DEFLATE text files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13857 ) Change subject: IMPALA-8549: Add support for scanning DEFLATE text files .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4755/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I45e41ab5a12637d396fef0812a09d71fa839b27a Gerrit-Change-Number: 13857 Gerrit-PatchSet: 11 Gerrit-Owner: Ethan Xue Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Ethan Xue Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 20:19:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14033 ) Change subject: IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/439/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I591e0fd736ea114aa52a999acf41806a94e49382 Gerrit-Change-Number: 14033 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 20:12:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14033 Change subject: IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection .. IMPALA-8837: [DOCS] HTTP support for proxy/delegation connection - Added a line on Knox support. Change-Id: I591e0fd736ea114aa52a999acf41806a94e49382 --- M docs/topics/impala_authentication.xml M docs/topics/impala_delegation.xml 2 files changed, 14 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/14033/1 -- To view, visit http://gerrit.cloudera.org:8080/14033 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I591e0fd736ea114aa52a999acf41806a94e49382 Gerrit-Change-Number: 14033 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4753/ -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 18:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14032 ) Change subject: WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046. .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4177/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia782bda3891dee1281cfcd5cf8817aa1226217ad Gerrit-Change-Number: 14032 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 17:42:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13880 ) Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4751/ -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 4 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 17:16:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13968 )
Change subject: IMPALA-8637: Implement transaction handling and locking for
ACID queries
..
Patch Set 10: Code-Review+2
(4 comments)
Thanks for upgrading the tests!
I have some nits, otherwise it is ok for me.
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py@105
PS10, Line 105: latest_open_txn
nit: I don't like the name too much, it made me assume that it is a txn id
instead of a timestamp. E.g. last_open_txn_start_time would be more descriptive.
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py@111
PS10, Line 111: "insert into {} values (sleep(20))".format(dummy_tbl))
nit: indent 4
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py@121
PS10, Line 121: self.client.cancel(handle)
Can we do it in a try: finally: block? If there is an exception for some reason
this long running query could affect some other tests that use metrics.
http://gerrit.cloudera.org:8080/#/c/13968/10/tests/query_test/test_acid.py@140
PS10, Line 140: _get_transactions
nit: _get_impala_transactions would be more precise
--
To view, visit http://gerrit.cloudera.org:8080/13968
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa
Gerrit-Change-Number: 13968
Gerrit-PatchSet: 10
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 07 Aug 2019 17:09:05 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 15: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 15 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 07 Aug 2019 17:01:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 15: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4754/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 15 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 07 Aug 2019 17:01:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 14 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 07 Aug 2019 17:01:00 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046.
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14032 Change subject: WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046. .. WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046. HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. This change sets 'engine' parameter to 'impala' in all the affected HMS API calls. For now this feature works with external tables only as COMPUTE STATS is not yet supported for managed tables (see IMPALA-8836). Change-Id: Ia782bda3891dee1281cfcd5cf8817aa1226217ad --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M tests/metadata/test_hms_integration.py 7 files changed, 108 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14032/1 -- To view, visit http://gerrit.cloudera.org:8080/14032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia782bda3891dee1281cfcd5cf8817aa1226217ad Gerrit-Change-Number: 14032 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 5: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/438/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:53:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Restrict bit unpacking to unsigned integer types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14031 ) Change subject: IMPALA-8796: Restrict bit unpacking to unsigned integer types .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4176/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cb57f43cc93e5dda6b6b2ff1617f1631f8cbe1 Gerrit-Change-Number: 14031 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 16:52:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4174/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:47:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 10: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4175/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:47:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Hello Greg Rahn, Tamas Mate, Lars Volker, Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, Csaba Ringhofer, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14021 to look at the new patch set (#5). Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala - Added impala_transactions.xml - Added impala_default_transactional_type.xml Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml A docs/topics/impala_default_transactional_type.xml A docs/topics/impala_transactions.xml 4 files changed, 674 insertions(+), 759 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/5 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/438/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:46:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml File docs/topics/impala_default_transactional_type.xml: http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@42 PS4, Line 42: The query option applies only when integrated with Hive : 3. > nit: This could be in the pre-requisite note/important field. (please see m Done http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@50 PS4, Line 50: The default table type is not transactional. > nit: this is a bit ambiguous, would be more clear something like: Done http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@54 PS4, Line 54: The default table type is the insert-only transactional : table. > nit: would be more clear something like: Done http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@60 PS4, Line 60: : If either table properties, transactional or : transactional_properties, are set, this query option is ignored. : : : : The DEFAULT_TRANSACTION_TYPE query option does not affect external, Kudu, : or HBase tables as these cannot be transactional. : > nit: These could be in a note or important block, so users are warned about The first paragraph is common in query options. I will put the second paragraph in a note. I do not want to overuse the Note/Important as it will lessen the effect. http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_transactions.xml File docs/topics/impala_transactions.xml: http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_transactions.xml@59 PS4, Line 59: Impala does not support changing transactional properties of tables. For example, you : cannot alter a transactional table to a non-transactional table. > nit: this could be in a note/important as well Done -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:45:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. Patch Set 2: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4752/ -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 16:22:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4173/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:16:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Restrict bit unpacking to unsigned integer types
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14031 Change subject: IMPALA-8796: Restrict bit unpacking to unsigned integer types .. IMPALA-8796: Restrict bit unpacking to unsigned integer types Restrict bit unpacking to the unsigned integer types uint8_t, uint16_t, uint32_t and uint64_t to simplify the interface of bit unpacking. Testing: Modified some tests that used bool or signed types for unpacking. Change-Id: I36cb57f43cc93e5dda6b6b2ff1617f1631f8cbe1 --- M be/src/exec/parquet/parquet-bool-decoder.h M be/src/util/bit-packing.cc M be/src/util/bit-packing.inline.h M be/src/util/bit-stream-utils-test.cc M be/src/util/bit-stream-utils.h 5 files changed, 42 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/14031/1 -- To view, visit http://gerrit.cloudera.org:8080/14031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I36cb57f43cc93e5dda6b6b2ff1617f1631f8cbe1 Gerrit-Change-Number: 14031 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 10: (2 comments) http://gerrit.cloudera.org:8080/#/c/13968/7/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13968/7/tests/query_test/test_acid.py@124 PS7, Line 124: def _latest_open_transaction(self): : max_start = 0 : for txn in self._get_transactions(): : start = txn['start_time'] : if start > max_start: : max_start = start : return max_start : : def _any_open_heartbeated_transaction_since(self, since_start_time): : for txn in self._get_transactions(): : if txn['state'] == 'OPEN': : start = txn['start_time'] : if start > My concern here is that as our experience is that HMS never aborts transact In the new PS I only care about transactions that were opened by Impala. Also, I search for the latest open transaction in the beginning and wait for the newly created transaction to send a heartbeat. http://gerrit.cloudera.org:8080/#/c/13968/9/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13968/9/tests/query_test/test_acid.py@104 PS9, Line 104: i > flake8: E265 block comment should start with '# ' Done -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:10:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#10). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. Testing: Added an exhaustive test that checks heartbeating. It creates a long running transaction with sleep(). While the transaction is executing the test periodically checks whether there is an OPEN transaction that has sent a heartbeat to HMS. If it found one then the test succeeds. TODOs: * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java M tests/query_test/test_acid.py 9 files changed, 530 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/10 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 10 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/13968/9/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13968/9/tests/query_test/test_acid.py@104 PS9, Line 104: # flake8: E265 block comment should start with '# ' -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:06:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#9). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. Testing: Added an exhaustive test that checks heartbeating. It creates a long running transaction with sleep(). While the transaction is executing the test periodically checks whether there is an OPEN transaction that has sent a heartbeat to HMS. If it found one then the test succeeds. TODOs: * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java M tests/query_test/test_acid.py 9 files changed, 530 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/9 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 9 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 4: (5 comments) Hi Alex, thank you for the doc update. Added some minor comments. http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml File docs/topics/impala_default_transactional_type.xml: http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@42 PS4, Line 42: The query option applies only when integrated with Hive : 3. nit: This could be in the pre-requisite note/important field. (please see my comment on line 60-68) http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@50 PS4, Line 50: The default table type is not transactional. nit: this is a bit ambiguous, would be more clear something like: Default value, the created table will not be transactional. http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@54 PS4, Line 54: The default table type is the insert-only transactional : table. nit: would be more clear something like: The created table will be marked as transactional, INSERT queries will be transactional on this table. http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_default_transactional_type.xml@60 PS4, Line 60: : If either table properties, transactional or : transactional_properties, are set, this query option is ignored. : : : : The DEFAULT_TRANSACTION_TYPE query option does not affect external, Kudu, : or HBase tables as these cannot be transactional. : nit: These could be in a note or important block, so users are warned about the prerequisites and scope of the query option. http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_transactions.xml File docs/topics/impala_transactions.xml: http://gerrit.cloudera.org:8080/#/c/14021/4/docs/topics/impala_transactions.xml@59 PS4, Line 59: Impala does not support changing transactional properties of tables. For example, you : cannot alter a transactional table to a non-transactional table. nit: this could be in a note/important as well -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 16:00:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8741: Speed up bit unpacking by vectorisation
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/13807 ) Change subject: IMPALA-8741: Speed up bit unpacking by vectorisation .. Patch Set 23: We found that the improvement in UnpackValues doesn't help much with dictionary and delta decoding, because dict decoding uses UnpackAndDecodeValues, which decodes the value right after unpacking as opposed to first unpacking all values and then decoding them. Delta is also faster this way. Therefore it is uncertain if introducing vectorisation is worth the added complexity now. On the other hand, restricting the unpacking output types to unsigned integers is independent from vectorisation and is a good idea, so I'll create a separate issue for it. -- To view, visit http://gerrit.cloudera.org:8080/13807 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9e452a547973778bbd8d768c608e1a32e948f947 Gerrit-Change-Number: 13807 Gerrit-PatchSet: 23 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 07 Aug 2019 15:58:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/13968/8/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13968/8/tests/query_test/test_acid.py@104 PS8, Line 104: # flake8: E265 block comment should start with '# ' -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 15:35:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#8). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. Testing: Added an exhaustive test that checks heartbeating. It creates a long running transaction via a debug action injects sleeps into the HdfsTableSink. While the transaction is executing the test periodically checks whether there is an OPEN transaction that has sent a heartbeat to HMS. If it found one then the test succeeds. TODOs: * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/exec/hdfs-table-sink.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java M tests/query_test/test_acid.py 10 files changed, 533 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/8 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 8 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4750/ -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 07 Aug 2019 15:34:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/437/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 15:11:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. IMPALA-8600: Fix test_acid_compaction Apparently Hive doesn't update the writeIds on a transactional table after compaction. This breaks an assumption made in the REFRESH table logic that only does an actual refresh when the HMS writeId is different than the one cached locally. As a result the non-partitioned ACID tables aren't refreshed when a REFRESH table is invoked in Impala right after a major compaction. Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Reviewed-on: http://gerrit.cloudera.org:8080/14027 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 9 insertions(+), 29 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 14:50:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Hello Gabor Kaszab, Attila Jeges, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13983 to look at the new patch set (#4). Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. IMPALA-7374: [DOCS] Document the new DATE data type in Impala Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 --- M docs/impala.ditamap M docs/topics/impala_date.xml 2 files changed, 153 insertions(+), 65 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13983/4 -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/437/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 14:46:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4171/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 14:05:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4172/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 14:05:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13968 )
Change subject: IMPALA-8637: Implement transaction handling and locking for
ACID queries
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13968/7/tests/query_test/test_acid.py
File tests/query_test/test_acid.py:
http://gerrit.cloudera.org:8080/#/c/13968/7/tests/query_test/test_acid.py@124
PS7, Line 124: def _any_open_heartbeated_transaction(self):
: hive_transactions = self.run_stmt_in_hive("SHOW
TRANSACTIONS")
: for transaction_line in hive_transactions.split('\n'):
: transaction_columns = transaction_line.split(',')
: if len(transaction_columns) < 4:
: continue
: transaction_state = transaction_columns[1]
: transaction_start_time = transaction_columns[2]
: transaction_last_heartbeat = transaction_columns[3]
: if transaction_state == 'OPEN':
: if transaction_start_time != transaction_last_heartbeat:
: return True
: return False
My concern here is that as our experience is that HMS never aborts transactions
due to missing heartbeats, so there can be quite old open transactions, and if
there was ever any heartbeat sent for them, they will satisfy this condition.
This seems possible to me for long running Hive inserts/compactions, so this
function can return true even if Impala does not send heartbeats at all.
I would do the following changes to make the test stronger:
- get the count or set of open_heartbeated transacations instead of just
checking their existence
- make the test sequential
- check the count/set before starting the query, and then expect it to be
increased/have a new transaction at one point
--
To view, visit http://gerrit.cloudera.org:8080/13968
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa
Gerrit-Change-Number: 13968
Gerrit-PatchSet: 7
Gerrit-Owner: Zoltan Borok-Nagy
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 07 Aug 2019 14:03:12 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4170/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:48:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4753/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:29:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4752/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:26:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Hello Yongzhi Chen, Gabor Kaszab, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13968 to look at the new patch set (#7). Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. IMPALA-8637: Implement transaction handling and locking for ACID queries Adds background thread to the Frontend that periodically heartbeats the opened transactions and locks. Most of the logic is implemented in the newly added TransactionKeepalive class. TransactionKeepalive keeps track of the creation time of the transactions and locks, and only heartbeat them if they are older then the sleep interval. This way we don't heartbeat short-running queries unnecessarily. Testing: Added an exhaustive test that checks heartbeating. It creates a long running transaction via a debug action injects sleeps into the HdfsTableSink. While the transaction is executing the test periodically checks whether there is an OPEN transaction that has sent a heartbeat to HMS. If it found one then the test succeeds. TODOs: * add metrics Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa --- M be/src/exec/hdfs-table-sink.cc M be/src/service/client-request-state.cc M be/src/service/frontend.cc M be/src/service/frontend.h M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/main/java/org/apache/impala/service/TransactionKeepalive.java M tests/query_test/test_acid.py 10 files changed, 516 insertions(+), 20 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/13968/7 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 7 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4169/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:26:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:25:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 13:18:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Daniel Becker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 2: It seems the build failed for unrelated reasons. -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 13:13:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Daniel Becker has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14029 ) Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) The actual DCHECK failure was fixed by the change for IMPALA-8833. Added a DCHECK to RleBatchDecoder so that it does not accept bit widths higher than the width of its type parameter. Also preventing UnpackAndDecodeValues from using higher bit widths than 32, the width of the dictionary index type, with a static assert. This also reduces compile time because the compiler does not have to generate code for invalid bit widths. Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 --- M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/dict-test.cc M be/src/util/rle-encoding.h 4 files changed, 13 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14029/2 -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 2 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13880 ) Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4751/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 4 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 13:07:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13880 ) Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 4 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 13:07:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7770: SPLIT PART to support negative indexes
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13880 ) Change subject: IMPALA-7770: SPLIT_PART to support negative indexes .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13880 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2db762989a90bd95661a59eb9c11a29eb2edfafb Gerrit-Change-Number: 13880 Gerrit-PatchSet: 3 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 13:07:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8840: Check failed: num bytes <= sizeof(T) (5 vs. 4)
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14029 Change subject: IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) .. IMPALA-8840: Check failed: num_bytes <= sizeof(T) (5 vs. 4) Added a DCHECK to RleBatchDecoder so that it does not accept bit widths higher than the width of its type parameter. Also preventing UnpackAndDecodeValues from using higher bit widths than 32, the width of its dictionary index type, with a static assert. Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 --- M be/src/util/bit-packing.h M be/src/util/bit-packing.inline.h M be/src/util/dict-test.cc M be/src/util/rle-encoding.h 4 files changed, 13 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/14029/1 -- To view, visit http://gerrit.cloudera.org:8080/14029 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I93461ba2cabb5ec7e0b65dcd62844fcbfa597d16 Gerrit-Change-Number: 14029 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Norbert Luksa has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. IMPALA-8755: Frontend support for Z-ordering Extended the SQL grammar with an optional flag for SORT BY, ZORDER. If set, the new 'sort.algorithm' table property will be set to ZORDER and the information will sink down to the backend. The default algorithm is indicated by LEXICAL and can be omitted. Examples are: CREATE TABLE t (a INT, b INT) PARTITIONED BY (c INT) SORT BY ZORDER (a, b); CREATE TABLE t SORT BY ZORDER (int_col,id) LIKE u; CREATE TABLE t LIKE PARQUET '/foo' SORT BY ZORDER (id,zip); ALTER TABLE t SORT BY ZORDER (int_col,id); The following two are the same queries: CREATE TABLE t (a INT, b INT) SORT BY (a, b); CREATE TABLE t (a INT, b INT) SORT BY LEXICAL (a, b); Currently ZORDER has the same functionality as a simple SORT BY clause, therefore hidden behind a feature flag: unlock_zorder. The custom sorting with Z-ordering will be in a different commit later. Testing: * Added tests for the ZORDER option for every SORT BY test. * Modified some tests by adding the LEXICAL option. * The .test workloads are temporarily put in separate test files in order to set up the feature flag. These tests are run from tests/custom_cluster/test_zorder.py which is a duplication of the relevant tests, but with CustomClusterTestSuite decorator. Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d --- M be/src/common/global-flags.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/DataSinks.thrift M common/thrift/JniCatalog.thrift M common/thrift/PlanNodes.thrift M common/thrift/Types.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/SortInfo.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/planner/AnalyticPlanner.java M fe/src/main/java/org/apache/impala/planner/ExchangeNode.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/planner/SortNode.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlTest.java M fe/src/test/java/org/apache/impala/analysis/ToSqlUtilsTest.java M fe/src/test/java/org/apache/impala/authorization/AuthorizationStmtTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java A testdata/workloads/functional-planner/queries/PlannerTest/insert-sort-by-zorder.test A testdata/workloads/functional-query/queries/QueryTest/alter-table-zorder.test A testdata/workloads/functional-query/queries/QueryTest/create-table-as-select-zorder.test A testdata/workloads/functional-query/queries/QueryTest/create-table-like-file-zorder.test A testdata/workloads/functional-query/queries/QueryTest/create-table-like-table-zorder.test A testdata/workloads/functional-query/queries/QueryTest/create-table-zorder.test A testdata/workloads/functional-query/queries/QueryTest/show-create-table-zorder.test M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test A tests/custom_cluster/test_zorder.py 44 files changed, 1,749 insertions(+), 146 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/55/13955/6 -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Norbert Luksa has posted comments on this change. (
http://gerrit.cloudera.org:8080/13955 )
Change subject: IMPALA-8755: Frontend support for Z-ordering
..
Patch Set 6:
(14 comments)
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java:
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java@58
PS5, Line 58: getRowFormat(),
HdfsFileFormat.fromThrift(getFileFormat()), compression, null,
> line too long (102 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2581
PS5, Line 2581: "tblproperties ('sort.algorithm'='ZORDER')", "Table
definition must not " +
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java
File fe/src/test/java/org/apache/impala/analysis/ParserTest.java:
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@2469
PS5, Line 2469: ParserError("ALTER TABLE TEST PARTITION (year=2009,
month=4) SORT BY ZORDER " +
> line too long (96 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/analysis/ParserTest.java@2629
PS5, Line 2629: // Create table like other table with zsort columns
> line too long (91 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/planner/PlannerTest.java
File fe/src/test/java/org/apache/impala/planner/PlannerTest.java:
http://gerrit.cloudera.org:8080/#/c/13955/5/fe/src/test/java/org/apache/impala/planner/PlannerTest.java@262
PS5, Line 262: // Add a test table with a SORT BY ZORDER clause to test
that the corresponding sort
> line too long (94 > 90)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py
File tests/custom_cluster/test_zorder.py:
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@18
PS5, Line 18: import shlex
> flake8: F401 'pytest' imported but unused
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@22
PS5, Line 22: from tests.common.custom_cluster_test_suite import
CustomClusterTestSuite
> flake8: F401 'copy.deepcopy' imported but unused
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@33
PS5, Line 33: VALID_SECTION_NAMES = ["CREATE_TABLE", "CREATE_VIEW", "QUERY",
"RESULTS"]
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@78
PS5, Line 78:
> flake8: E501 line too long (97 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@81
PS5, Line 81: @
> flake8: E303 too many blank lines (2)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@90
PS5, Line 90: # definition. The table is created, then the output of "SHOW
CREATE TABLE" is used to
> flake8: E303 too many blank lines (3)
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@94
PS5, Line 94: # Properties to filter before comparing results
> flake8: E302 expected 2 blank lines, found 3
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@254
PS5, Line 254: RESULTS_DB_NAME_TOKEN = "show_create_table_test_db"
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/13955/5/tests/custom_cluster/test_zorder.py@302
PS5, Line 302:
> flake8: W391 blank line at end of file
Done
--
To view, visit http://gerrit.cloudera.org:8080/13955
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d
Gerrit-Change-Number: 13955
Gerrit-PatchSet: 6
Gerrit-Owner: Norbert Luksa
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Norbert Luksa
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 07 Aug 2019 12:37:16 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 11:30:22 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4750/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 07 Aug 2019 11:22:28 + Gerrit-HasComments: No
[Impala-ASF-CR] Bump CDP BUILD NUMBER to 1318335
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14005 ) Change subject: Bump CDP_BUILD_NUMBER to 1318335 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I432e8c02dbd349a3507bfabfef2727914537652c Gerrit-Change-Number: 14005 Gerrit-PatchSet: 4 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 07 Aug 2019 11:22:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 10:42:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4749/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 10:42:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 10:34:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14027 ) Change subject: IMPALA-8600: Fix test_acid_compaction .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4167/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 10:06:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/13983/3/docs/topics/impala_date.xml File docs/topics/impala_date.xml: http://gerrit.cloudera.org:8080/#/c/13983/3/docs/topics/impala_date.xml@79 PS3, Line 79: Hive 3.1 and lower versions > Done Did you forget to upload that patch set? :) -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 07 Aug 2019 09:31:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8600: Fix test acid compaction
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14027 Change subject: IMPALA-8600: Fix test_acid_compaction .. IMPALA-8600: Fix test_acid_compaction Apparently Hive doesn't update the writeIds on a transactional table after compaction. This breaks an assumption made in the REFRESH table logic that only does an actual refresh when the HMS writeId is different than the one cached locally. As a result the non-partitioned ACID tables aren't refreshed when a REFRESH table is invoked in Impala right after a major compaction. Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 --- M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java 1 file changed, 9 insertions(+), 29 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/14027/1 -- To view, visit http://gerrit.cloudera.org:8080/14027 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I58b79f8864b31e18eca818032ad5a9af954913f6 Gerrit-Change-Number: 14027 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] IMPALA-8637: Implement transaction handling and locking for ACID queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13968 ) Change subject: IMPALA-8637: Implement transaction handling and locking for ACID queries .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4748/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13968 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iaa37899b24aa114be642bf8772b4e0f882865cfa Gerrit-Change-Number: 13968 Gerrit-PatchSet: 6 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 07 Aug 2019 07:25:20 + Gerrit-HasComments: No
