[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has uploaded a new patch set (#11) to the change originally created by Sudhanshu Arora. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. acid: Filter unwanted files based on ACID state. - Added new functionality in AcidUtils to filter out files in uncommitted directories, and to find the latest valid base data and filter out files corresponding to older deltas or bases. - Changed Table loading to only load writeIds for transactional tables, and enabled a previously-ignored unit test. - Modified Hive configuration to enable support for compactions: -- Need to pass Tez on the HMS classpath, since HMS actually schedules compactions rather than HS2. -- Had to configure a worker thread for the compactor, or else compactions wouldn't proceed even when manually triggered. Testing: - New unit tests (AcidUtilsTest) for filtering logic. - New e2e test to read data written by Hive in an insert-only table, with INSERT, INSERT OVERWRITE, and compaction. Also tests negative cases e2e. To enable the e2e test, this adds support for a 'HIVE_QUERY' section to the test script files. To make it reasonably fast, this uses Thrift to connect to HS2 rather than shelling out to beeline. In order for this to work properly, a bit of extra special-casing had to be added to the test utility. This commit was co-authored by Sudhanshu Arora and Todd Lipcon. Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e --- M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java A fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M fe/src/test/resources/hive-site.xml.py M testdata/bin/run-hive-server.sh A testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test A testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/acid.test M tests/common/impala_connection.py M tests/common/impala_test_suite.py A tests/query_test/test_acid.py M tests/util/test_file_parser.py 20 files changed, 743 insertions(+), 175 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13334/11 -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 11 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/13311/13/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java: http://gerrit.cloudera.org:8080/#/c/13311/13/fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java@540 PS13, Line 540: "Table functional_orc_def.full_transactional_table not supported. Transactional (ACID)" + line too long (95 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:03:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 11: (1 comment) http://gerrit.cloudera.org:8080/#/c/13334/11/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13334/11/tests/common/impala_test_suite.py@332 PS11, Line 332: l flake8: E741 ambiguous variable name 'l' -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 11 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:04:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has uploaded a new patch set (#13) to the change originally created by Sudhanshu Arora. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. IMPALA-8435. Prohibit operations on transactional table. Copied some code from Hive to identify if the table is transactional, insert only table. Also modified code to prohibit write operations on insert only table. That code will be reverted once we add support for write operations on insert only table. Testing Done: - Added a new test in AnalyzerTest Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java A fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/bin/generate-schema-statements.py M testdata/datasets/README M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv 16 files changed, 296 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/13311/13 -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 11: Made some changes/fixes and updated the commit message. Take a look and let me know what you think. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 11 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:03:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4298/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:14:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13382 ) Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4297/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 07:13:54 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4299/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 12 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:14:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 16: (2 comments) http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py File tests/hooks/test_hooks.py: http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py@33 PS16, Line 33: from tests.common.file_utils import assert_file_in_dir_contains,\ flake8: F401 'tests.common.file_utils.assert_no_files_in_dir_contain' imported but unused http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py@131 PS16, Line 131: flake8: E251 unexpected spaces around keyword / parameter equals -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 07:16:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Hello Andrew Sherman, Anonymous Coward (498), Austin Nobis, Fredy Wijaya, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13352 to look at the new patch set (#16). Change subject: IMPALA-8473: publish lineage info via hook .. IMPALA-8473: publish lineage info via hook This commit introduces a hook mechanism for publishing, lineage data specifically, but query information more generally, from Impala. The legacy behavior of writing the lineage file is being retained but deprecated. Hooks can be implemented by downstream consumers (i.e. runtime dependencies) to hook into supported places during Impala query execution: - impalad startup - post-query execution The consumers are to be frontend Java dependencies intiated at runtime. 2 backend flags configure this behavior: - `query_exec_hook_classes` specifies a comma-separated list of hook consumer implementation classes that are instantiated and registered at impala start up. - `num_query_exec_hook_threads` specifies the number of threads to use for asynchronous hook execution. (Relevant if multiple hooks are registered.) Lineage information is passed from the backend after a query completes (but before it returns) and given to every hook to execute asynchronously. IOW, a query may complete and return to the user before any or all hooks have completed executing. An exception during hook post-query execution will simply be logged and will not bring down the system. Tests: - added unit tests for FE hook execution - added e2e tests for hook configuration, execution, error Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift A fe/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java A fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java A fe/src/main/java/org/apache/impala/hooks/QueryExecHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/hooks/QueryExecHookManagerTest.java A fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/CountingQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/DummyQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/PostQueryErrorExecHook.java A tests/hooks/test_hooks.py 19 files changed, 1,118 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13352/16 -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13352 )
Change subject: IMPALA-8473: publish lineage info via hook
..
Patch Set 14:
(1 comment)
As discussed in Slack, I've uploaded a simple change from the last patchset
where we pass the json lineage string as opposed to the thrift object. (This
is actually the same approach as the first few patchsets
http://gerrit.cloudera.org:8080/#/c/13352/15/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13352/15/be/src/service/impala-server.cc@499
PS15, Line 499: if (AreQueryHooksEnabled()) {
> One thing to be aware of this is that LogLineageRecord() is called from Unr
Really great point; I've copied this comment into the hook interface javadoc
--
To view, visit http://gerrit.cloudera.org:8080/13352
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Gerrit-Change-Number: 13352
Gerrit-PatchSet: 14
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 22 May 2019 07:18:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] (WIP) IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13400 ) Change subject: (WIP) IMPALA-8507: Support DROP TABLE statement with Kudu/HMS integration .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3323/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13400 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d3b93957cc66009ad7a67fc513be2068f156abc Gerrit-Change-Number: 13400 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 07:18:07 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3325/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 11 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 07:59:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 16: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 08:09:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 08:02:25 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 12: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4299/ -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 12 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 09:05:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/13353 ) Change subject: IMPALA-8443: Record time spent in authorization in the runtime profile .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13353/2/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java: http://gerrit.cloudera.org:8080/#/c/13353/2/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@426 PS2, Line 426: (Ranger|Sentry) > Yeah in my CR: https://gerrit.cloudera.org/c/13309/6/fe/src/main/java/org/a Thanks guys for looking into it. Austin, a new class would be ideal, but I could not find anything else to refactor with the analyzeAndAuthorize and it seems it would have too many boilerplate parts alone. The best option as you mentioned exposing the AuthorizationConfig. Fredy, I assume we should pass the timeline to the AuthorizationChecker and it could be called next to the logging. The EventSequence should be modified that when it marks an event it returns the timestamp as well. So, I should wait for https://gerrit.cloudera.org/#/c/13309/ to be submitted for now. Is my understanding correct? -- To view, visit http://gerrit.cloudera.org:8080/13353 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae Gerrit-Change-Number: 13353 Gerrit-PatchSet: 2 Gerrit-Owner: Tamas Mate Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Wed, 22 May 2019 09:17:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13382 ) Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4297/ -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 12:28:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 14: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4298/ -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 12:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504: Introduce the new Kudu storage handler
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13358 ) Change subject: IMPALA-8504: Introduce the new Kudu storage handler .. Patch Set 5: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13358 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I75bcd5246005f4e35251aef9219f4d07eeb87dc6 Gerrit-Change-Number: 13358 Gerrit-PatchSet: 5 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Adar Dembo Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Andrew Wong Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 13:01:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/13375 ) Change subject: IMPALA-8504 (part 2): Support CREATE TABLE statement with Kudu/HMS integration .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13375/1/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/13375/1/fe/src/main/java/org/apache/impala/service/KuduCatalogOpExecutor.java@102 PS1, Line 102: String tableId = table.getTableId(); > I think we should avoid that, as without HMS integration enabled, it is har True, sounds good. -- To view, visit http://gerrit.cloudera.org:8080/13375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icffe412395f47f5e07d97bad457020770cfa7502 Gerrit-Change-Number: 13375 Gerrit-PatchSet: 1 Gerrit-Owner: Hao Hao Gerrit-Reviewer: Grant Henke Gerrit-Reviewer: Hao Hao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 13:00:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13402 Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests "hive -e SQL..." without further parameters no longer works when USE_CDP_HIVE=true (it doesn't establish a connection). Some tests used this to load data. part2: there were some places that still called hive without a beeline connection. Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b --- M tests/metadata/test_views_compatibility.py M tests/query_test/test_scanners.py 2 files changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13402/1 -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3327/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 15:23:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8443: Record time spent in authorization in the runtime profile
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/13353 )
Change subject: IMPALA-8443: Record time spent in authorization in the runtime
profile
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13353/2/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/13353/2/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@426
PS2, Line 426: (Ranger|Sentry)
> Thanks guys for looking into it.
We can put the timeline in the AuthorizationContext and it will be available in
the postAuthorize(). It looks something like this.
@Override
public void postAuthorize(AuthorizationContext authzCtx) {
long durationMs = authzCtx.getTimeline().markEvent("Authorization
finished");
LOG.debug("Authorization check took {} ms", durationMs);
}
Yeah, I think it's best to wait till this CR is merged. Hopefully I can get a
+2 soon :)
--
To view, visit http://gerrit.cloudera.org:8080/13353
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5bb85e57fcc75d41f3eb2911e6d375e0da6f82ae
Gerrit-Change-Number: 13353
Gerrit-PatchSet: 2
Gerrit-Owner: Tamas Mate
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tamas Mate
Gerrit-Comment-Date: Wed, 22 May 2019 15:33:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8450: Add support for zstd and lz4 in parquet
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13396 )
Change subject: IMPALA-8450: Add support for zstd and lz4 in parquet
..
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/service/query-options.cc@761
PS6, Line 761: clevel, ZSTD_maxCLevel()));
nit: could be moved to the previous line
http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h
File be/src/util/codec.h:
http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h@63
PS6, Line 63:private:
: friend class Codec;
: friend class HdfsParquetTableWriter;
: friend void TestCompression(int, int, int,
THdfsCompression::type);
I would prefer to make the members public. If you want to keep private members,
than this should be a "class" instead of "struct".
http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h@68
PS6, Line 68: clevel_
Can you mention in a comment that currently only ZSTD uses this?
+ Using a longer name like compression_level_ would make the purpose clearer.
http://gerrit.cloudera.org:8080/#/c/13396/6/cmake_modules/FindZstd.cmake
File cmake_modules/FindZstd.cmake:
http://gerrit.cloudera.org:8080/#/c/13396/6/cmake_modules/FindZstd.cmake@36
PS6, Line 36:
nit: whitespace consistency
http://gerrit.cloudera.org:8080/#/c/13396/6/tests/query_test/test_insert_parquet.py
File tests/query_test/test_insert_parquet.py:
http://gerrit.cloudera.org:8080/#/c/13396/6/tests/query_test/test_insert_parquet.py@144
PS6, Line 144: test_insert_parquet_multi_codecs
I would prefer to move most of the logic to a .test file.
An example .test file that creates tables with different query options:
https://github.com/apache/impala/blob/master/testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test
http://gerrit.cloudera.org:8080/#/c/13396/6/tests/query_test/test_insert_parquet.py@159
PS6, Line 159: self.execute_query("set COMPRESSION_CODEC=ZSTD")
: insert_stmt = """insert into {0} values
: (6,false,8.99815,'2004-01-26
05:00:31.75','2004-01-26','Winter is coming'
: ,'~Dracarys!!~'),
: (7,false,9.9,'1999-09-09
09:00:59.99','1999-09-09','I am Iron Man'
: ,'I am inevitable - <>#$%@*(#$^%*@$#'),
: (8,NULL,NULL,NULL,NULL,NULL,NULL)
The test could be probably shorter by CTAS-ing/inserting from alltypestiny, e.g
"insert into {0} select * from functional.alltypestiny where id = 1 or id = 2",
and compering the two tables at the end. Note that some types (DATE) + NULLs
are missing from alltypestiny.
--
To view, visit http://gerrit.cloudera.org:8080/13396
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I98c6dcf3d0a873380e4fa4cf03eb7e924e4ee768
Gerrit-Change-Number: 13396
Gerrit-PatchSet: 6
Gerrit-Owner: Abhishek Rawat
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 22 May 2019 15:49:10 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Code-Review+2 Ouch, I only grepped for "hive, not 'hive before https://gerrit.cloudera.org/#/c/13282/ -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 16:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 16:12:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Code-Review-1 Seems like Hive3 is more strict about SQL syntax. Need to modify some statements in views-compatibility.test -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 16:28:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 1: Verified-1 -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 16:28:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 16: (3 comments) I added a few comments based on recent changes http://gerrit.cloudera.org:8080/#/c/13352/16//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13352/16//COMMIT_MSG@38 PS16, Line 38: to every hook to execute asynchronously. IOW, a Nit: spell out IOW to help future non-native English speakers http://gerrit.cloudera.org:8080/#/c/13352/16/fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java File fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java: http://gerrit.cloudera.org:8080/#/c/13352/16/fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java@64 PS16, Line 64:* The query gets unregistered, lineage record gets logged It may be clearer to have a TODO here like TODO(Impala-) (where Impala- is a real Jira), then the work to fix this can be tracked. http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py File tests/hooks/test_hooks.py: http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py@299 PS16, Line 299: shutil.rmtree(self.LOG_DIR, ignore_errors=True) If you make LOG_DIR unique you could leave it around which might help someone with debugging -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 16 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 16:44:45 + Gerrit-HasComments: Yes
[native-toolchain-CR] Patch Thrift to 0.9.3-p6 to eliminate erroneous ssl warnings.
Thomas Marshall has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13383 ) Change subject: Patch Thrift to 0.9.3-p6 to eliminate erroneous ssl warnings. .. Patch Thrift to 0.9.3-p6 to eliminate erroneous ssl warnings. This will be used to fix "IMPALA-8333: Remove Impala Shell warnings part 2". Remove misleading deprecation warning from the TSSLSocket.validate method. This method is used to determine if the peer's SSL certificate should be validated. The deprecation warning message is a mistake, probably as the result of a confusion of the role of the validate method with the use of a deprecated parameter, also named validate, that is used in the initializer. This code was removed in subsequent Thrift releases. TESTING: I built the toolchain locally and using jenkins. I built impala with the new toolchain and tested that IMPALA-8333 is fixed. Change-Id: I84c2de4e6e3b21a4e3a5591f469d47ab34101687 Reviewed-on: http://gerrit.cloudera.org:8080/13383 Reviewed-by: Thomas Marshall Tested-by: Thomas Marshall --- M buildall.sh A source/thrift/thrift-0.9.3-patches/0006-remove-misleading-deprecation-warning.patch 2 files changed, 29 insertions(+), 0 deletions(-) Approvals: Thomas Marshall: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I84c2de4e6e3b21a4e3a5591f469d47ab34101687 Gerrit-Change-Number: 13383 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Thomas Marshall
[native-toolchain-CR] Patch Thrift to 0.9.3-p6 to eliminate erroneous ssl warnings.
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/13383 ) Change subject: Patch Thrift to 0.9.3-p6 to eliminate erroneous ssl warnings. .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13383 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I84c2de4e6e3b21a4e3a5591f469d47ab34101687 Gerrit-Change-Number: 13383 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Thomas Marshall Gerrit-Comment-Date: Wed, 22 May 2019 16:52:24 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/13334 )
Change subject: acid: Filter unwanted files based on ACID state.
..
Patch Set 12:
(3 comments)
Gone through the first half. Will continue tomorrow.
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java
File fe/src/main/java/org/apache/impala/util/AcidUtils.java:
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@48
PS12, Line 48: private static final Pattern BASE_PATTERN =
Pattern.compile("base_(\\d+)(/.*)?");
: private static final Pattern DELTA_PATTERN =
: Pattern.compile("delta_(\\d+)_(\\d+)(_\\d+)?(/.*)?");
nit: maybe you could use "name-capturing groups" in the regex to make it more
verbose, e.g. "delta_(?\\d+)..."
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@151
PS12, Line 151: statementGroup.substring(1)
DELTA_PATTERN allows strings like 'delta_1_2_asdf', i.e. we have the trailing
'(/.*)?' group, but no statement id. In that case it will throw an exception.
Name-capturing groups can also help here.
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@170
PS12, Line 170: >=
Shouldn't this be '=='? Can it happen that baseNum is greater than max base id?
--
To view, visit http://gerrit.cloudera.org:8080/13334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e
Gerrit-Change-Number: 13334
Gerrit-PatchSet: 12
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 22 May 2019 16:58:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6903: Download profile from WebUI in text format
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/1 ) Change subject: IMPALA-6903: Download profile from WebUI in text format .. Patch Set 6: (1 comment) Overall, LGTM. Just one more minor comment. http://gerrit.cloudera.org:8080/#/c/1/3/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/1/3/tests/webserver/test_web_pages.py@551 PS3, Line 551: try: > I need no timeout. actually, since this is just waiting for the query to finish. you can just use self.client.execute(query), right? -- To view, visit http://gerrit.cloudera.org:8080/1 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie030c2bb330211f51840417b9f7880f19174af7b Gerrit-Change-Number: 1 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 17:14:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Todd Lipcon has removed a vote on this change. Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13382 ) Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Patch Set 5: Tim, can you take another look at this? I had to make a couple changes to deal with HBase tables (turns out the HBase row key is considered a clustering column but does need stats computed) -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 17:26:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 14: Verified+1 Test failures are due to the unrelated Java issue hitting every build. -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 17:29:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has removed a vote on this change. Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Removed Verified-1 by Impala Public Jenkins -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Sudhanshu Arora has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/13334/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/13334/8/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@571 PS8, Line 571: to > nit: camel case Done http://gerrit.cloudera.org:8080/#/c/13334/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/13334/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@549 PS8, Line 549: > keeping this as a Predicate makes sense to me, since this is common Done http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java File fe/src/main/java/org/apache/impala/util/AcidUtils.java: http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@151 PS12, Line 151: statementGroup.substring(1) > DELTA_PATTERN allows strings like 'delta_1_2_asdf', i.e. we have the traili delta_1_2_asdf will not match the pattern. delta_1_2/_asdf will match. However group3 in this case will be null so we will not try to get a long value. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 12 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 17:30:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13402 to look at the new patch set (#2). Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests "hive -e SQL..." without further parameters no longer works when USE_CDP_HIVE=true (it doesn't establish a connection). Some tests used this to load data. part2: there were some places that still called hive without a beeline connection. I had to break up views-compatibility.test for the different versions of Hive. Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b --- A testdata/workloads/functional-query/queries/QueryTest/views-compatibility-hive2-only.test A testdata/workloads/functional-query/queries/QueryTest/views-compatibility-hive3-only.test M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test M tests/metadata/test_views_compatibility.py M tests/query_test/test_scanners.py 5 files changed, 58 insertions(+), 31 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13402/2 -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13382 ) Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Patch Set 5: Verified+1 Overriding the Jenkins failure - seems the precommits are failing with some unrelated Java errors. -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 17:25:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py File tests/metadata/test_views_compatibility.py: http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py@157 PS2, Line 157: e flake8: E722 do not use bare except' http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py@157 PS2, Line 157: flake8: E261 at least two spaces before inline comment -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 17:33:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Fredy Wijaya has uploaded this change for review. (
http://gerrit.cloudera.org:8080/13399
Change subject: IMPALA-8564: Add table/view create time in the lineage graph
..
IMPALA-8564: Add table/view create time in the lineage graph
This patch adds table/view create time in the lineage graph. This is
needed for Impala/Atlas integration. See ATLAS-3080.
Below is an example of the updated lineage graph.
{
"queryText":"create table lineage_test_tbl as select int_col, tinyint_col
from functional.alltypes",
"queryId":"0:0",
"hash":"407f23b24758ffcb2ac445b9703f5c44",
"user":"dummy_user",
"timestamp":1547867921,
"edges":[
{
"sources":[
1
],
"targets":[
0
],
"edgeType":"PROJECTION"
},
{
"sources":[
3
],
"targets":[
2
],
"edgeType":"PROJECTION"
}
],
"vertices":[
{
"id":0,
"vertexType":"COLUMN",
"vertexId":"int_col",
"metadata":{
"createTime":1558546457,
"tableName":"default.lineage_test_tbl"
}
},
{
"id":1,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.int_col",
"metadata":{
"createTime":1557860978,
"tableName":"functional.alltypes"
}
},
{
"id":2,
"vertexType":"COLUMN",
"vertexId":"tinyint_col",
"metadata":{
"createTime":-1,
"tableName":"default.lineage_test_tbl"
}
},
{
"id":3,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.tinyint_col",
"metadata":{
"createTime":1557860978,
"tableName":"functional.alltypes"
}
}
]
}
Testing:
- Updated lineage tests in PlannerTest
- Updated test_lineage.py
- Ran all FE tests
Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/lineage-util.h
M common/thrift/Frontend.thrift
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M tests/custom_cluster/test_lineage.py
15 files changed, 2,150 insertions(+), 619 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13399/7
--
To view, visit http://gerrit.cloudera.org:8080/13399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
Gerrit-Change-Number: 13399
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Sudhanshu Arora has posted comments on this change. (
http://gerrit.cloudera.org:8080/13311 )
Change subject: IMPALA-8435. Prohibit operations on transactional table.
..
Patch Set 14: Code-Review+1
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13311/9/testdata/bin/generate-schema-statements.py
File testdata/bin/generate-schema-statements.py:
http://gerrit.cloudera.org:8080/#/c/13311/9/testdata/bin/generate-schema-statements.py@249
PS9, Line 249:
> typo
Done
http://gerrit.cloudera.org:8080/#/c/13311/9/testdata/bin/generate-schema-statements.py@255
PS9, Line 255: tblproperties = {}
> I think I'd feel better about this successive-split thing if we did the spl
Done
http://gerrit.cloudera.org:8080/#/c/13311/9/testdata/bin/generate-schema-statements.py@260
PS9, Line 260: ')
> (I think my uneasiness about this code is that we end up with a somewhat mi
Done
--
To view, visit http://gerrit.cloudera.org:8080/13311
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a
Gerrit-Change-Number: 13311
Gerrit-PatchSet: 14
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Comment-Date: Wed, 22 May 2019 18:06:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 18:09:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has removed a vote on this change. Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Removed Code-Review-2 by Todd Lipcon -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. IMPALA-8435. Prohibit operations on transactional table. Copied some code from Hive to identify if the table is transactional, insert only table. Also modified code to prohibit write operations on insert only table. That code will be reverted once we add support for write operations on insert only table. Testing Done: - Added a new test in AnalyzerTest Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Reviewed-on: http://gerrit.cloudera.org:8080/13311 Tested-by: Todd Lipcon Reviewed-by: Sudhanshu Arora Reviewed-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java A fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/bin/generate-schema-statements.py M testdata/datasets/README M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv 16 files changed, 296 insertions(+), 15 deletions(-) Approvals: Todd Lipcon: Looks good to me, approved; Verified Sudhanshu Arora: Looks good to me, but someone else must approve -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 15 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8435. Prohibit operations on transactional table.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/13311 ) Change subject: IMPALA-8435. Prohibit operations on transactional table. .. Patch Set 14: Code-Review-2 -- To view, visit http://gerrit.cloudera.org:8080/13311 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I740dc4ce0dbbc0c2e042b01832e606cc1ac4132a Gerrit-Change-Number: 13311 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 18:09:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3329/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 18:30:14 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Sudhanshu Arora has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 12: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 12 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 18:31:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8546: collect logs from docker containers
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13393 Change subject: IMPALA-8546: collect logs from docker containers .. IMPALA-8546: collect logs from docker containers This modifies containers to put logs in /opt/impala/logs, then mounts that directory to $IMPALA_HOME/logs/.../ so that logs will be collected on the host and scooped up by jenkins jobs. The layout of the log directory is a little different to the non-dockerised containers because I wanted to avoid sharing log directories between containers. Change-Id: I24bcaa521882d450d43d1f2ca34767e7ce36bbd2 --- M bin/start-impala-cluster.py M docker/catalogd/Dockerfile M docker/coord_exec/Dockerfile M docker/coordinator/Dockerfile M docker/executor/Dockerfile M docker/impala_base/Dockerfile M docker/statestored/Dockerfile 7 files changed, 15 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13393/3 -- To view, visit http://gerrit.cloudera.org:8080/13393 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I24bcaa521882d450d43d1f2ca34767e7ce36bbd2 Gerrit-Change-Number: 13393 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13334 )
Change subject: acid: Filter unwanted files based on ACID state.
..
Patch Set 12:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java
File fe/src/main/java/org/apache/impala/util/AcidUtils.java:
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@48
PS12, Line 48: private static final Pattern BASE_PATTERN =
Pattern.compile("base_(\\d+)(/.*)?");
: private static final Pattern DELTA_PATTERN =
: Pattern.compile("delta_(\\d+)_(\\d+)(_\\d+)?(/.*)?");
> nit: maybe you could use "name-capturing groups" in the regex to make it mo
Done. Also changed the capturing a bit so we only capture the numeric portion
of the optional statement ID
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/util/AcidUtils.java@170
PS12, Line 170: >=
> Shouldn't this be '=='? Can it happen that baseNum is greater than max base
Done
--
To view, visit http://gerrit.cloudera.org:8080/13334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e
Gerrit-Change-Number: 13334
Gerrit-PatchSet: 12
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 22 May 2019 18:44:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has uploaded a new patch set (#13) to the change originally created by Sudhanshu Arora. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. acid: Filter unwanted files based on ACID state. - Added new functionality in AcidUtils to filter out files in uncommitted directories, and to find the latest valid base data and filter out files corresponding to older deltas or bases. - Changed Table loading to only load writeIds for transactional tables, and enabled a previously-ignored unit test. - Modified Hive configuration to enable support for compactions: -- Need to pass Tez on the HMS classpath, since HMS actually schedules compactions rather than HS2. -- Had to configure a worker thread for the compactor, or else compactions wouldn't proceed even when manually triggered. Testing: - New unit tests (AcidUtilsTest) for filtering logic. - New e2e test to read data written by Hive in an insert-only table, with INSERT, INSERT OVERWRITE, and compaction. Also tests negative cases e2e. To enable the e2e test, this adds support for a 'HIVE_QUERY' section to the test script files. To make it reasonably fast, this uses Thrift to connect to HS2 rather than shelling out to beeline. In order for this to work properly, a bit of extra special-casing had to be added to the test utility. This commit was co-authored by Sudhanshu Arora and Todd Lipcon. Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e --- M fe/pom.xml M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java A fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M fe/src/test/resources/hive-site.xml.py M testdata/bin/run-hive-server.sh A testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test A testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/acid.test M tests/common/impala_connection.py M tests/common/impala_test_suite.py A tests/query_test/test_acid.py M tests/util/test_file_parser.py 20 files changed, 743 insertions(+), 175 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13334/13 -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has uploaded a new patch set (#14) to the change originally created by Sudhanshu Arora. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. acid: Filter unwanted files based on ACID state. - Added new functionality in AcidUtils to filter out files in uncommitted directories, and to find the latest valid base data and filter out files corresponding to older deltas or bases. - Changed Table loading to only load writeIds for transactional tables, and enabled a previously-ignored unit test. - Modified Hive configuration to enable support for compactions: -- Need to pass Tez on the HMS classpath, since HMS actually schedules compactions rather than HS2. -- Had to configure a worker thread for the compactor, or else compactions wouldn't proceed even when manually triggered. Testing: - New unit tests (AcidUtilsTest) for filtering logic. - New e2e test to read data written by Hive in an insert-only table, with INSERT, INSERT OVERWRITE, and compaction. Also tests negative cases e2e. To enable the e2e test, this adds support for a 'HIVE_QUERY' section to the test script files. To make it reasonably fast, this uses Thrift to connect to HS2 rather than shelling out to beeline. In order for this to work properly, a bit of extra special-casing had to be added to the test utility. This commit was co-authored by Sudhanshu Arora and Todd Lipcon. Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e --- M fe/pom.xml M fe/src/compat-hive-2/java/org/apache/hadoop/hive/common/ValidWriteIdList.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/main/java/org/apache/impala/util/AcidUtils.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java A fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java M fe/src/test/resources/hive-site.xml.py M testdata/bin/run-hive-server.sh A testdata/workloads/functional-query/queries/QueryTest/acid-compaction.test A testdata/workloads/functional-query/queries/QueryTest/acid-negative.test A testdata/workloads/functional-query/queries/QueryTest/acid.test M tests/common/impala_connection.py M tests/common/impala_test_suite.py A tests/query_test/test_acid.py M tests/util/test_file_parser.py 21 files changed, 744 insertions(+), 176 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/13334/14 -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/13334/13/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13334/13/tests/common/impala_test_suite.py@332 PS13, Line 332: l flake8: E741 ambiguous variable name 'l' -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 18:46:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 14: (1 comment) http://gerrit.cloudera.org:8080/#/c/13334/14/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/13334/14/tests/common/impala_test_suite.py@332 PS14, Line 332: l flake8: E741 ambiguous variable name 'l' -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 18:47:10 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 22 May 2019 19:00:44 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 13: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3332/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 13 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 19:22:37 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/13334 )
Change subject: acid: Filter unwanted files based on ACID state.
..
Patch Set 14:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
File fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java:
http://gerrit.cloudera.org:8080/#/c/13334/12/fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java@206
PS12, Line 206: loadedFds_ =
AcidUtils.filterFilesForAcidState(loadedFds_,
: relativeDirPaths);
nit: fits to one line
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java
File fe/src/main/java/org/apache/impala/util/AcidUtils.java:
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@48
PS14, Line 48: (?:/.*)?
I have deltas like this produced by compaction:
base_002_v108
+Same as line 54.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@54
PS14, Line 54:// Optional path suffix.
:"(?:/.*)?");
Is this optional? We should only match files inside directories as far as I
understand.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@105
PS14, Line 105: public boolean test(String dirPath) {
To support upgraded tables this should also accept any path without "/", so
where it is a file directly inside the table/partition directory.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@170
PS14, Line 170: if (baseWriteId == maxBaseWriteId) {
: validDescriptors.add(fd);
We will go to this branch with any file if getMaxBaseId() didn't found any base
directory, so maxBaseWriteId == Long.MIN_VALUE. This is not necessarily a
problem, as we should accept all valid deltas in this case, but it seems more
logical to add them in the next branch.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@182
PS14, Line 182: // If the table was not transaction table then return all
fds.
: if (validDescriptors.isEmpty()) {
: return fileDescriptors;
: }
Can't we enter this if the table was truncated, but the old deltas are still
there?
--
To view, visit http://gerrit.cloudera.org:8080/13334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e
Gerrit-Change-Number: 13334
Gerrit-PatchSet: 14
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 22 May 2019 19:28:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13402 ) Change subject: IMPALA-8524: part2: Avoid calling "hive" via command line in EE tests .. Patch Set 2: Code-Review+2 (2 comments) http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py File tests/metadata/test_views_compatibility.py: http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py@81 PS2, Line 81: nit: needs +2 indentation http://gerrit.cloudera.org:8080/#/c/13402/2/tests/metadata/test_views_compatibility.py@84 PS2, Line 84: unique_database) nit: needs +2 indentation -- To view, visit http://gerrit.cloudera.org:8080/13402 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia45b64cc1da78190e6f239a5f462308d7fa56f4b Gerrit-Change-Number: 13402 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 19:39:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8546: collect logs from docker containers
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13393 ) Change subject: IMPALA-8546: collect logs from docker containers .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3331/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13393 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I24bcaa521882d450d43d1f2ca34767e7ce36bbd2 Gerrit-Change-Number: 13393 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 19:44:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Anurag Mantripragada, Vihang Karajgaonkar, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12940 to look at the new patch set (#5). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: After run a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/query_test/test_observability.py 7 files changed, 127 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/12940/5 -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Patch Set 1: This builds on my machine. Running tests to make sure upstream Jenkins jobs work -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 19:46:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/12940/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/12940/5/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@930 PS5, Line 930: final Timer storageLdTimer = getMetrics().getTimer(Table.STORAGE_MD_LOAD_DURATION_METRIC); line too long (98 > 90) http://gerrit.cloudera.org:8080/#/c/12940/5/fe/src/main/java/org/apache/impala/catalog/Table.java File fe/src/main/java/org/apache/impala/catalog/Table.java: http://gerrit.cloudera.org:8080/#/c/12940/5/fe/src/main/java/org/apache/impala/catalog/Table.java@140 PS5, Line 140: public static final String STORAGE_MD_LOAD_DURATION_METRIC = "storage-metadata-load-duration"; line too long (96 > 90) -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 19:46:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4300/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 19:46:24 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Patch Set 1: Code-Review+2 > Patch Set 1: > > This builds on my machine. Running tests to make sure upstream Jenkins jobs > work -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 19:47:28 + Gerrit-HasComments: No
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13334 ) Change subject: acid: Filter unwanted files based on ACID state. .. Patch Set 14: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks// : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e Gerrit-Change-Number: 13334 Gerrit-PatchSet: 14 Gerrit-Owner: Sudhanshu Arora Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 22 May 2019 19:48:44 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13403 Change subject: Remove hwx.public.repo maven repo .. Remove hwx.public.repo maven repo The hwx.public.repo (nexus-private.hortonworks.com) contains jars that conflict with impala.cdh.repo (GBN based s3 bucket). This means that Impala can build and test with a mix of jars from both. The latest symptom of this is frontend tests failing with messages like: java.lang.NoClassDefFoundError: org/apache/hadoop/security/ssl/OpenSSLSocketFactory$SSLChannelMode This happens because hadoop-aws is on a more recent version than hadoop-common. Since Impala builds without the hwx.public.repo, let's remove it. Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 --- M impala-parent/pom.xml 1 file changed, 0 insertions(+), 8 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/13403/1 -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Hello Bharath Vissapragada, Anurag Mantripragada, Vihang Karajgaonkar, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12940 to look at the new patch set (#6). Change subject: IMPALA-7322: Add storage wait time to profile .. IMPALA-7322: Add storage wait time to profile Add metrics to record storage wait time for operations with metadata load in catalog for hdfs, kudu and hbase tables. Pass storage wait time from catalog to fe through thrift and log total storage load time in query profile. Testing: Ran queries that can trigger all of, none of or some of the related tables loading. Check query profile for each query. Check catalog metrics for each table. Add unit tests to test_observability.py Ran all core tests. Sample output: After run a hbase query (Metadata load finished is divided into several lines because of limitation of commit message): Query Compilation: 4s401ms - Metadata load started: 661.084us (661.084us) - Metadata load finished. loaded-tables=1/1 load-requests=1 catalog-updates=3 storage-load-time=233ms: 3s819ms (3s819ms) - Analysis finished: 3s820ms (763.979us) - Value transfer graph computed: 3s820ms (63.193us) Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HBaseTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/KuduTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M tests/query_test/test_observability.py 7 files changed, 129 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/12940/6 -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 17: (11 comments) http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py File tests/hooks/test_hooks.py: http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@30 PS17, Line 30: from getpass import getuser flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@31 PS17, Line 31: from ImpalaService import ImpalaHiveServer2Service flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@32 PS17, Line 32: from TCLIService import TCLIService flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@33 PS17, Line 33: from thrift.transport.TSocket import TSocket flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@34 PS17, Line 34: from thrift.transport.TTransport import TBufferedTransport flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@35 PS17, Line 35: from thrift.protocol import TBinaryProtocol flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@36 PS17, Line 36: from tests.common.custom_cluster_test_suite import CustomClusterTestSuite flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@37 PS17, Line 37: from tests.common.file_utils import assert_file_in_dir_contains,\ flake8: E402 module level import not at top of file http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@37 PS17, Line 37: from tests.common.file_utils import assert_file_in_dir_contains,\ flake8: F401 'tests.common.file_utils.assert_no_files_in_dir_contain' imported but unused http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@217 PS17, Line 217: class TestHooksStartupFail(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@248 PS17, Line 248: d flake8: E304 blank lines found after function decorator -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 20:02:13 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Hello Andrew Sherman, Anonymous Coward (498), Austin Nobis, Fredy Wijaya, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13352 to look at the new patch set (#17). Change subject: IMPALA-8473: publish lineage info via hook .. IMPALA-8473: publish lineage info via hook This commit introduces a hook mechanism for publishing, lineage data specifically, but query information more generally, from Impala. The legacy behavior of writing the lineage file is being retained but deprecated. Hooks can be implemented by downstream consumers (i.e. runtime dependencies) to hook into supported places during Impala query execution: - impalad startup - post-query execution The consumers are to be frontend Java dependencies intiated at runtime. 2 backend flags configure this behavior: - `query_exec_hook_classes` specifies a comma-separated list of hook consumer implementation classes that are instantiated and registered at impala start up. - `num_query_exec_hook_threads` specifies the number of threads to use for asynchronous hook execution. (Relevant if multiple hooks are registered.) Lineage information is passed from the backend after a query completes (but before it returns) and given to every hook to execute asynchronously. In other words, a query may complete and return to the user before any or all hooks have completed executing. An exception during hook post-query execution will simply be logged and will not bring down the system. Tests: - added unit tests for FE hook execution - added e2e tests for hook configuration, execution, error - ran full build, tests Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift A fe/src/main/java/org/apache/impala/hooks/PostQueryHookContext.java A fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java A fe/src/main/java/org/apache/impala/hooks/QueryExecHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/hooks/QueryExecHookManagerTest.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/CountingQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/DummyQueryExecHook.java A fe/src/test/java/org/apache/impala/testutil/PostQueryErrorExecHook.java A tests/hooks/test_hooks.py 20 files changed, 1,183 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13352/17 -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13352 )
Change subject: IMPALA-8473: publish lineage info via hook
..
Patch Set 17:
(15 comments)
still one more iteration needed
http://gerrit.cloudera.org:8080/#/c/13352/16//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13352/16//COMMIT_MSG@38
PS16, Line 38: to every hook to execute asynchronously. In other words,
> Nit: spell out IOW to help future non-native English speakers
good point; done
http://gerrit.cloudera.org:8080/#/c/13352/17/fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java
File fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java:
http://gerrit.cloudera.org:8080/#/c/13352/17/fe/src/main/java/org/apache/impala/hooks/QueryExecHook.java@25
PS17, Line 25: public interface QueryExecHook {
Before we merge this, are we happy with this name? The backend seems to use
the terminology "query event", e.g. `(*request_state)->query_events()`
https://github.com/apache/impala/blob/3.2.0/be/src/service/impala-server.cc#L934
Perhaps we should rename this to `QueryEventHook` and likewise in other similar
places.
http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py
File tests/hooks/test_hooks.py:
http://gerrit.cloudera.org:8080/#/c/13352/16/tests/hooks/test_hooks.py@299
PS16, Line 299: pass
> If you make LOG_DIR unique you could leave it around which might help someo
much better; done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py
File tests/hooks/test_hooks.py:
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@30
PS17, Line 30: from getpass import getuser
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@31
PS17, Line 31: from ImpalaService import ImpalaHiveServer2Service
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@32
PS17, Line 32: from TCLIService import TCLIService
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@33
PS17, Line 33: from thrift.transport.TSocket import TSocket
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@34
PS17, Line 34: from thrift.transport.TTransport import TBufferedTransport
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@35
PS17, Line 35: from thrift.protocol import TBinaryProtocol
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@36
PS17, Line 36: from tests.common.custom_cluster_test_suite import
CustomClusterTestSuite
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@37
PS17, Line 37: from tests.common.file_utils import assert_file_in_dir_contains,\
> flake8: E402 module level import not at top of file
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@37
PS17, Line 37: from tests.common.file_utils import assert_file_in_dir_contains,\
> flake8: F401 'tests.common.file_utils.assert_no_files_in_dir_contain' impor
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@205
PS17, Line 205: def __wait_for_file(self, filepath, timeout_s=10):
Should this go in common/file_utils.py? I didn't put it there now because it's
just a first-pass, quick-and-dirty implementation
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@217
PS17, Line 217: class TestHooksStartupFail(CustomClusterTestSuite):
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/13352/17/tests/hooks/test_hooks.py@248
PS17, Line 248: d
> flake8: E304 blank lines found after function decorator
Done
--
To view, visit http://gerrit.cloudera.org:8080/13352
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Gerrit-Change-Number: 13352
Gerrit-PatchSet: 17
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 22 May 2019 20:11:33 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3334/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 20:31:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Fredy Wijaya has uploaded a new patch set (#8). (
http://gerrit.cloudera.org:8080/13399 )
Change subject: IMPALA-8564: Add table/view create time in the lineage graph
..
IMPALA-8564: Add table/view create time in the lineage graph
This patch adds table/view create time in the lineage graph. This is
needed for Impala/Atlas integration. See ATLAS-3080.
Below is an example of the updated lineage graph.
{
"queryText":"create table lineage_test_tbl as select int_col, tinyint_col
from functional.alltypes",
"queryId":"0:0",
"hash":"407f23b24758ffcb2ac445b9703f5c44",
"user":"dummy_user",
"timestamp":1547867921,
"edges":[
{
"sources":[
1
],
"targets":[
0
],
"edgeType":"PROJECTION"
},
{
"sources":[
3
],
"targets":[
2
],
"edgeType":"PROJECTION"
}
],
"vertices":[
{
"id":0,
"vertexType":"COLUMN",
"vertexId":"int_col",
"metadata":{
"createTime":1558546457,
"tableName":"default.lineage_test_tbl"
}
},
{
"id":1,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.int_col",
"metadata":{
"createTime":1557860978,
"tableName":"functional.alltypes"
}
},
{
"id":2,
"vertexType":"COLUMN",
"vertexId":"tinyint_col",
"metadata":{
"createTime":1558546457,
"tableName":"default.lineage_test_tbl"
}
},
{
"id":3,
"vertexType":"COLUMN",
"vertexId":"functional.alltypes.tinyint_col",
"metadata":{
"createTime":1557860978,
"tableName":"functional.alltypes"
}
}
]
}
Testing:
- Updated lineage tests in PlannerTest
- Updated test_lineage.py
- Ran all FE tests
Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/lineage-util.h
M common/thrift/Frontend.thrift
M common/thrift/LineageGraph.thrift
M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java
M fe/src/main/java/org/apache/impala/analysis/CreateOrAlterViewStmtBase.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
M testdata/workloads/functional-planner/queries/PlannerTest/lineage.test
M tests/custom_cluster/test_lineage.py
15 files changed, 2,150 insertions(+), 619 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13399/8
--
To view, visit http://gerrit.cloudera.org:8080/13399
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82
Gerrit-Change-Number: 13399
Gerrit-PatchSet: 8
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] acid: Filter unwanted files based on ACID state.
Todd Lipcon has posted comments on this change. (
http://gerrit.cloudera.org:8080/13334 )
Change subject: acid: Filter unwanted files based on ACID state.
..
Patch Set 14:
(4 comments)
I think Sudhanshu's AFK this afternoon so will try to rev the patch in his
absence.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java
File fe/src/main/java/org/apache/impala/util/AcidUtils.java:
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@48
PS14, Line 48: (?:/.*)?
> I have deltas like this produced by compaction:
Good point. I'll see if I can add a test case for this one.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@54
PS14, Line 54:// Optional path suffix.
:"(?:/.*)?");
> Is this optional? We should only match files inside directories as far as I
yea, I guess in contrast to base patterns, the existence of an empty delta
directory doesn't actually matter, so we could skip over the directories and
only match on the files. That said, I don't think there's harm in matching the
directories and it seems less surprising if this regex matches the dir (same as
above)
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@105
PS14, Line 105: public boolean test(String dirPath) {
> To support upgraded tables this should also accept any path without "/", so
Good point. Mind if I add a TODO to support upgraded tables? I'm not sure how
to test this at the moment and this patch is already large.
http://gerrit.cloudera.org:8080/#/c/13334/14/fe/src/main/java/org/apache/impala/util/AcidUtils.java@182
PS14, Line 182: // If the table was not transaction table then return all
fds.
: if (validDescriptors.isEmpty()) {
: return fileDescriptors;
: }
> Can't we enter this if the table was truncated, but the old deltas are stil
Yea, I think that's true. Unfortunately right now due to
https://issues.apache.org/jira/browse/HIVE-20137 and
https://issues.apache.org/jira/browse/HIVE-21750 there doesn't seem to be a way
to test this case at the moment. I'll fix it anyway and add a unit test.
--
To view, visit http://gerrit.cloudera.org:8080/13334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf0aeb36e10c827ead59ed7f67e731199394fe8e
Gerrit-Change-Number: 13334
Gerrit-PatchSet: 14
Gerrit-Owner: Sudhanshu Arora
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sudhanshu Arora
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Yongzhi Chen
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Wed, 22 May 2019 20:34:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3335/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 20:47:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3336/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 17 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 20:47:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8560: Prometheus metrics support in Impala
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/13345 )
Change subject: IMPALA-8560: Prometheus metrics support in Impala
..
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/13345/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13345/4//COMMIT_MSG@13
PS4, Line 13:
> I have local prometheus running on same node that is ingesting this metrics
Can you add a "Testing" section to the commit message? That'll make it more
visible for others.
http://gerrit.cloudera.org:8080/#/c/13345/5/be/src/util/metrics.h
File be/src/util/metrics.h:
http://gerrit.cloudera.org:8080/#/c/13345/5/be/src/util/metrics.h@112
PS5, Line 112: std::string ConvertToPrometheusSecs(std::stringstream* val,
TUnit::type unit) {
I have two issues with the function/pattern:
* The name of the function is confusing, since it's called for all metrics,
including those that don't need conversion and aren't time types. I think it
should be something that makes it more obvious at the callsite why it's called
and what it will do (I'd suggest a name but I don't think we've decided on the
behaviour yet).
* Converting to a string and back again is really awkward and hard to reason
about. I'm guessing this was a workaround for C++ template hell around T,
because T can be std::string which doesn't support a / operation.
We can solve the template thing in a cleaner way by having multiple
implementations of a templated function, with the non-numeric ones just no-ops
(since we don't expect the conversion to be done for non-numeric metrics).
As a proof of concept I got it working for one metric - see
https://github.com/timarmstrong/impala/tree/prometheus
http://gerrit.cloudera.org:8080/#/c/13345/5/be/src/util/metrics.h@191
PS5, Line 191: TIMS_MS
TIME_MS
--
To view, visit http://gerrit.cloudera.org:8080/13345
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5349085a2007b568cb97f9b8130804ea64d7bb08
Gerrit-Change-Number: 13345
Gerrit-PatchSet: 5
Gerrit-Owner: Harshil
Gerrit-Reviewer: Harshil
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 22 May 2019 20:44:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint
Hello Bharath Vissapragada, Michael Ho, Sudhanshu Arora, Mike Yoder, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13299 to look at the new patch set (#3). Change subject: IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint .. IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint This patch provides an option to use HTTP based transport for HiveServer2 endpoint on coordinators that the clients can connect to query. HTTP(S) also works when external TLS is enabled using --ssl_server_certificate. Implemented only for HS2 compatible thrift server since, unlike beeswax, its session management does not need to be tied to the underlying TCP conneciton. Thirft's http transport is modified to support BASIC authentication via ldap. For convenience of developing and reviewing, this patch is based on another that copied THttpServer and THttpTransport into Impala's codebase. TODO = - Fuzz test the http endpoint - Add tests for LDAP + HTTPS Testing === - Parameterized JdbcTest and LdapJdbcTest to work for HS2 + HTTP mode - Added LdapHS2Test, which directly calls into the Hiveserver2 interface using a thrift http client. Manual testing with Beeline client (from Apache Hive), which has builtin support to connect to HTTP(S) based HS2 compatible endpoints. Example -- HTTP mode: > start-impala-cluster.py > JDBC_URL="jdbc:hive2://localhost:/default;transportMode=http" > beeline -u "$JDBC_URL" -- HTTPS mode: > cd $IMPALA_HOME > SSL_ARGS="--ssl_client_ca_certificate=./be/src/testutil/server-cert.pem \ --ssl_server_certificate=./be/src/testutil/server-cert.pem \ --ssl_private_key=./be/src/testutil/server-key.pem --hostname=localhost" > start-impala-cluster.py --impalad_args="$SSL_ARGS" \ --catalogd_args="$SSL_ARGS" --state_store_args="$SSL_ARGS" - Create a local trust store using 'keytool' and import the certificate from server-cert.pem (./clientkeystore in the example). > JDBC_URL="jdbc:hive2://localhost:/default;ssl=true;sslTrustStore= \ ./clientkeystore;trustStorePassword=password;transportMode=http" > beeline -u "$JDBC_URL" -- BASIC Auth with LDAP: > LDAP_ARGS="--enable_ldap_auth --ldap_uri='ldap://...' \ --ldap_bind_pattern='...' --ldap_passwords_in_clear_ok" > start-impala-cluster.py --impalad_args="$LDAP_ARGS" > JDBC_URL="jdbc:hive2://localhost:28000/default;user=...;password=\ ...;transportMode=http" > beeline -u "$JDBC_URL" -- HTTPS mode with LDAP: > start-impala-cluster.py --impalad_args="$LDAP_ARGS $SSL_ARGS" \ --catalogd_args="$SSL_ARGS" --state_store_args="$SSL_ARGS" > JDBC_URL="jdbc:hive2://localhost:28000/default;user=...;password=\ ...;ssl=true;sslTrustStore=./clientkeystore;trustStorePassword=\ password;transportMode=http" > beeline -u "$JDBC_URL" Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff --- M be/src/rpc/auth-provider.h M be/src/rpc/authentication-test.cc M be/src/rpc/authentication.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M bin/start-impala-cluster.py M common/thrift/generate_error_codes.py M common/thrift/metrics.json A fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java M fe/src/test/java/org/apache/impala/customcluster/LdapJdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTest.java M fe/src/test/java/org/apache/impala/service/JdbcTestBase.java M fe/src/test/java/org/apache/impala/testutil/ImpalaJdbcClient.java M fe/src/test/resources/users.ldif M tests/common/impala_cluster.py 24 files changed, 641 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13299/3 -- To view, visit http://gerrit.cloudera.org:8080/13299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff Gerrit-Change-Number: 13299 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mike Yoder Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3
Hello Michael Ho, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13298 to look at the new patch set (#2). Change subject: IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3 .. IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3 This is a mechanical change that just copies several files over from thrift. This is for convenience in reviewing changes to these files, which have been submitted as a follow up patch. Change-Id: I1916e17eaeb7854eb93c2415396f0ee0243e4e32 --- M be/src/transport/CMakeLists.txt A be/src/transport/THttpServer.cpp A be/src/transport/THttpServer.h A be/src/transport/THttpTransport.cpp A be/src/transport/THttpTransport.h 5 files changed, 602 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/13298/2 -- To view, visit http://gerrit.cloudera.org:8080/13298 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1916e17eaeb7854eb93c2415396f0ee0243e4e32 Gerrit-Change-Number: 13298 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/13298 )
Change subject: IMPALA-8538 (part 1) Copied THttp(Server|Transport) from
thrift-0.9.3
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13298/2/be/src/transport/THttpServer.h
File be/src/transport/THttpServer.h:
http://gerrit.cloudera.org:8080/#/c/13298/2/be/src/transport/THttpServer.h@56
PS2, Line 56: virtual boost::shared_ptr
getTransport(boost::shared_ptr trans) {
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/13298
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1916e17eaeb7854eb93c2415396f0ee0243e4e32
Gerrit-Change-Number: 13298
Gerrit-PatchSet: 2
Gerrit-Owner: Thomas Marshall
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Comment-Date: Wed, 22 May 2019 20:57:06 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12940 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3337/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6dde7e394b7c1c396d835ef6aa0a55930c0a8660 Gerrit-Change-Number: 12940 Gerrit-PatchSet: 6 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 22 May 2019 20:51:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8564: Add table/view create time in the lineage graph
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13399 ) Change subject: IMPALA-8564: Add table/view create time in the lineage graph .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3338/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13399 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If4f578d7b299a76c30323b10a883ba32f8713d82 Gerrit-Change-Number: 13399 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 20:59:16 + Gerrit-HasComments: No
[Impala-ASF-CR] Add decimal parquet to TableFormatInfo.KNOWN FILE FORMATS
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/13398 ) Change subject: Add decimal_parquet to TableFormatInfo.KNOWN_FILE_FORMATS .. Patch Set 2: > Patch Set 2: > > Does this end up increasing the exhaustive test runtime substantially by > adding yet another dimension? Or is the dimension generation not driven from > this constant? > > If so, do we think this is necessary/useful coverage? I'm honestly not sure, so I'm running an exhaustive build now to confirm. But my understanding is that this constant won't affect that. If you git grep for KNOWN_FILE_FORMATS, it shows up in very few places. The context of this patch is that I tried to run the workload runner against Impala on a deployed cluster, and I'd loaded decimal parquet data ahead of time. However, the convention (admittedly a bit arbitrarily) is that db's are named __. The error I'm seeing is... 17:10:01 Traceback (most recent call last): 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/bin/run-workload.py", line 246, in 17:10:01 workload_runners.append(WorkloadRunner(workload, scale_factor, config)) 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/tests/performance/workload_runner.py", line 73, in __init__ 17:10:01 self._generate_test_vectors() 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/tests/performance/workload_runner.py", line 90, in _generate_test_vectors 17:10:01 self._test_vectors.append(TableFormatInfo.create_from_string(dataset, tf)) 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/tests/common/test_dimensions.py", line 80, in create_from_string 17:10:01 compression_type=compression_type) 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/tests/common/test_dimensions.py", line 42, in __init__ 17:10:01 self.__validate() 17:10:01 File "/data/jenkins/workspace/Quasar-Executor/Impala-workload-runner/tests/common/test_dimensions.py", line 46, in __validate 17:10:01 raise ValueError, 'Unknown file format: %s' % self.file_format 17:10:01 ValueError: Unknown file format: decimal_parquet -- To view, visit http://gerrit.cloudera.org:8080/13398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9fc9c0ec82318e740a06493c1b4e4719c7162db Gerrit-Change-Number: 13398 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 20:59:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint
Thomas Marshall has posted comments on this change. (
http://gerrit.cloudera.org:8080/13299 )
Change subject: IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server
endpoint
..
Patch Set 3:
(17 comments)
http://gerrit.cloudera.org:8080/#/c/13299/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13299/2//COMMIT_MSG@21
PS2, Line 21:
:
: TODO
> It seems to me like it would be easier to just copy-paste this, since the T
Sure. I checked, and the differences between this version and thrift master's
THttp* is quite minimal. Its certainly less work for me not to have to patch
native-toolchain.
I'm interested in trying to contribute this upstream, but I assume that process
will take awhile, so for this patch I'll go ahead and take the "[not for
review]" off of the patch that copies the files that this patch is based on.
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc
File be/src/rpc/authentication.cc:
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@502
PS2, Line 502: LOG(ERROR) << "Failed to decode base64 auth string from: "
> for all of the log messages in this function, can we include the remote soc
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@505
PS2, Line 505: }
> stack allocating here seems quite dangerous without constraining the length
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@507
PS2, Line 507: if (colon == std::string::npos) {
> Looking at Base64Decode, it doesn't seem to null-terminate the output, but
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@521
PS2, Line 521:
> again I'd feel safer about the above code if we used C++ strings, like:
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@957
PS2, Line 957:
> this function is a bit weirdly named, since in the HTTP case, it isn't sett
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@977
PS2, Line 977: switch (underlying_transport_type) {
> can we just LOG(FATAL) here since it would be a coding bug?
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/rpc/authentication.cc@998
PS2, Line 998: }
> same
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/service/impala-server.cc@2424
PS2, Line 2424: ThriftServer* server;
> Maybe we should use '-1' to mean disable? port 0 usually means "use an ephe
The problem with that is that the equivalent flags, eg. for hs2_port and
beewax, use 0 to mean disabled. I'm not sure its a good idea either to have
inconsistent behavior between flags or to change the behavior of the existing
flags.
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.h
File be/src/transport/THttpServer.h:
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.h@71
PS2, Line 71: Wraps a transport i
> that's a bit of an odd choice of type instead of std::string
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.h@80
PS2, Line 80:
> explicit
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp
File be/src/transport/THttpServer.cpp:
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp@68
PS2, Line 68: rwarded
> probably needs to be made case-insensitive (odd that x-forwarded-for is not
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp@69
PS2, Line 69:
> probably need to also check that sz >= 7, otherwise we might read past the
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp@76
PS2, Line 76: }
: authorized_ = true;
> is base64AuthString_ were a string you could just use a simple assignment h
Done
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp@79
PS2, Line 79: }
> I think it's worth considering this a bit carefully. It seems that the desi
As discussed, I went with a design where we set the username on the connection
context each time we authenticate and then check that the session username
matches the current connection username when doing operations. This works
because each connection corresponds to a single thread which will read some
headers and process the corresponding rpc one at a time.
One concern this leaves me with is that now we're hitting ldap on every rpc. If
that seems like a potential perf issue, I could add a list of already
authenticated base64 strings here and check that before calling the auth fn, or
I could add support for cookies, which is probably a better solution but I'm
not sure how much work it would be.
http://gerrit.cloudera.org:8080/#/c/13299/2/be/src/transport/THttpServer.cpp@193
PS2,
[Impala-ASF-CR] IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13298 ) Change subject: IMPALA-8538 (part 1) Copied THttp(Server|Transport) from thrift-0.9.3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3339/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13298 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1916e17eaeb7854eb93c2415396f0ee0243e4e32 Gerrit-Change-Number: 13298 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Wed, 22 May 2019 21:31:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Hello Andrew Sherman, Anonymous Coward (498), Austin Nobis, Fredy Wijaya, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13352 to look at the new patch set (#18). Change subject: IMPALA-8473: publish lineage info via hook .. IMPALA-8473: publish lineage info via hook This commit introduces a hook mechanism for publishing, lineage data specifically, but query information more generally, from Impala. The legacy behavior of writing the lineage file is being retained but deprecated. Hooks can be implemented by downstream consumers (i.e. runtime dependencies) to hook into supported places during Impala query execution: - impalad startup - query completion - see IMPALA-8572 for caveat/details The consumers are to be frontend Java dependencies intiated at runtime. 2 backend flags configure this behavior: - `query_event_hook_classes` specifies a comma-separated list of hook consumer implementation classes that are instantiated and registered at impala start up. - `query_event_hook_nthreads` specifies the number of threads to use for asynchronous hook execution. (Relevant if multiple hooks are registered.) Lineage information is passed from the backend after a query completes (but before it returns) and given to every hook to execute asynchronously. In other words, a query may complete and return to the user before any or all hooks have completed executing. An exception during hook on-query-complete execution will simply be logged and will not be (directly) fatal to the system. Tests: - added unit tests for FE hook execution - added e2e tests for hook configuration, execution, error - ran full build, tests Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift A fe/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java A fe/src/main/java/org/apache/impala/hooks/QueryEventHook.java A fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/CountingQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/PostQueryErrorEventHook.java A tests/hooks/test_hooks.py 20 files changed, 1,230 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13352/18 -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 18: (2 comments) http://gerrit.cloudera.org:8080/#/c/13352/18/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13352/18/be/src/service/impala-server.cc@266 PS18, Line 266: "QueryEventHook execution. If this number is >1 then hooks will execute concurrently. " line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/13352/18/be/src/service/impala-server.cc@506 PS18, Line 506: const Status& status = exec_env_->frontend()->CallQueryCompleteHooks(query_complete_context); line too long (97 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 21:36:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Hello Andrew Sherman, Anonymous Coward (498), Austin Nobis, Fredy Wijaya, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13352 to look at the new patch set (#19). Change subject: IMPALA-8473: publish lineage info via hook .. IMPALA-8473: publish lineage info via hook This commit introduces a hook mechanism for publishing, lineage data specifically, but query information more generally, from Impala. The legacy behavior of writing the lineage file is being retained but deprecated. Hooks can be implemented by downstream consumers (i.e. runtime dependencies) to hook into supported places during Impala query execution: - impalad startup - query completion - see IMPALA-8572 for caveat/details The consumers are to be frontend Java dependencies intiated at runtime. 2 backend flags configure this behavior: - `query_event_hook_classes` specifies a comma-separated list of hook consumer implementation classes that are instantiated and registered at impala start up. - `query_event_hook_nthreads` specifies the number of threads to use for asynchronous hook execution. (Relevant if multiple hooks are registered.) Lineage information is passed from the backend after a query completes (but before it returns) and given to every hook to execute asynchronously. In other words, a query may complete and return to the user before any or all hooks have completed executing. An exception during hook on-query-complete execution will simply be logged and will not be (directly) fatal to the system. Tests: - added unit tests for FE hook execution - added e2e tests for hook configuration, execution, error - ran full build, tests Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 --- M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift A fe/src/main/java/org/apache/impala/hooks/QueryCompleteContext.java A fe/src/main/java/org/apache/impala/hooks/QueryEventHook.java A fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java A fe/src/test/java/org/apache/impala/testutil/AlwaysErrorQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/CountingQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/DummyQueryEventHook.java A fe/src/test/java/org/apache/impala/testutil/PostQueryErrorEventHook.java A tests/hooks/test_hooks.py 20 files changed, 1,231 insertions(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/13352/19 -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
radford nguyen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 19: Think this is ready for final review now. Sorry about all the patchsets; don't really have my impala workflow established yet. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 21:39:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7290: part 2: Add HS2 support to Impala shell
Thomas Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/12884 ) Change subject: IMPALA-7290: part 2: Add HS2 support to Impala shell .. Patch Set 18: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/12884/16/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/12884/16/common/thrift/ImpalaService.thrift@545 PS16, Line 545: // Same as HS2 CloseOperation but can return additional information. > Weirdly the number of modified rows was returned in the operation handle: h Interesting. Well anyways, I think this is probably fine for the purposes of this patch. If we need better vanilla hs2 compatibility in the future for some reason we can always see about upstreaming some of this -- To view, visit http://gerrit.cloudera.org:8080/12884 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d5cc83d545aacc659523f29b1d6feed672e2a12 Gerrit-Change-Number: 12884 Gerrit-PatchSet: 18 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 22 May 2019 21:54:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13299 ) Change subject: IMPALA-8538: HS2 + HTTP(S) + BASIC/LDAP based thrift server endpoint .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3340/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13299 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic5569ac62ef3af2868b5d0581f5029dac736b2ff Gerrit-Change-Number: 13299 Gerrit-PatchSet: 3 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Mike Yoder Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 21:55:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 18: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3341/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 18 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 22:21:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13352 ) Change subject: IMPALA-8473: publish lineage info via hook .. Patch Set 19: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3342/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13352 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1 Gerrit-Change-Number: 13352 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Anonymous Coward (498) Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Wed, 22 May 2019 22:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] Add decimal parquet to TableFormatInfo.KNOWN FILE FORMATS
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13398 ) Change subject: Add decimal_parquet to TableFormatInfo.KNOWN_FILE_FORMATS .. Patch Set 2: Code-Review+1 This isn't referenced outside of the file - it's purely used for input validation. The actual data sets and workloads are driven by testdata/workloads/ and testdata/datasets We could consider just removing the validation if we're OK with it failing later on when it can't find the data set definition and/or database and/or tables. But this seems fine to unblock. Another possible solution would be to skip the validation when TableFormatInfo is created from tests/performance/workload_runner.py -- To view, visit http://gerrit.cloudera.org:8080/13398 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9fc9c0ec82318e740a06493c1b4e4719c7162db Gerrit-Change-Number: 13398 Gerrit-PatchSet: 2 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 22:38:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8473: publish lineage info via hook
Andrew Sherman has posted comments on this change. (
http://gerrit.cloudera.org:8080/13352 )
Change subject: IMPALA-8473: publish lineage info via hook
..
Patch Set 19: Code-Review+1
(3 comments)
I left a few comments but this looks OK as work in progress.
http://gerrit.cloudera.org:8080/#/c/13352/18/fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
File fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java:
http://gerrit.cloudera.org:8080/#/c/13352/18/fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java@40
PS18, Line 40: public class QueryEventHookManagerTest {
Please add a quick javadoc even if it seems obvious to you now
http://gerrit.cloudera.org:8080/#/c/13352/18/fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java@44
PS18, Line 44: private QueryCompleteContext mockQueryCompleteContext;
I don't understand why this is marked as Mock, it seems a simple class that can
be created easily in a test.
In general I don't find mockito tests easy to read and probably I am missing
something.
http://gerrit.cloudera.org:8080/#/c/13352/18/fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java@67
PS18, Line 67: private static QueryEventHookManager createSut(int nThreads,
String... hooks)
What is SUT? Maybe use something clearer to the reader
--
To view, visit http://gerrit.cloudera.org:8080/13352
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23a896537a98bfef07fb27c70e9a87c105cd77a1
Gerrit-Change-Number: 13352
Gerrit-PatchSet: 19
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Anonymous Coward (498)
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Wed, 22 May 2019 22:41:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/13382 ) Change subject: IMPALA-8215, IMPALA-8458. Fix setting stats without setting NDVs in local-catalog mode .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13382 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0b95de22954c7ad6715143fc42a1506289c095f Gerrit-Change-Number: 13382 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Wed, 22 May 2019 22:39:55 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Patch Set 1: Verified+1 This got past the build/dataload/fe_tests for both USE_CDP_HIVE=false and USE_CDP_HIVE=true. If it was going to fail, it would have failed by now. This is blocking lots of code changes, so +1 verifying. -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 22 May 2019 22:59:39 + Gerrit-HasComments: No
[Impala-ASF-CR] Remove hwx.public.repo maven repo
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13403 ) Change subject: Remove hwx.public.repo maven repo .. Remove hwx.public.repo maven repo The hwx.public.repo (nexus-private.hortonworks.com) contains jars that conflict with impala.cdh.repo (GBN based s3 bucket). This means that Impala can build and test with a mix of jars from both. The latest symptom of this is frontend tests failing with messages like: java.lang.NoClassDefFoundError: org/apache/hadoop/security/ssl/OpenSSLSocketFactory$SSLChannelMode This happens because hadoop-aws is on a more recent version than hadoop-common. Since Impala builds without the hwx.public.repo, let's remove it. Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Reviewed-on: http://gerrit.cloudera.org:8080/13403 Reviewed-by: Fredy Wijaya Tested-by: Joe McDonnell --- M impala-parent/pom.xml 1 file changed, 0 insertions(+), 8 deletions(-) Approvals: Fredy Wijaya: Looks good to me, approved Joe McDonnell: Verified -- To view, visit http://gerrit.cloudera.org:8080/13403 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6f1cf7efcad522c2df228c8c2b8068bb3002e407 Gerrit-Change-Number: 13403 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8450: Add support for zstd and lz4 in parquet
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/13396 ) Change subject: IMPALA-8450: Add support for zstd and lz4 in parquet .. Patch Set 7: (56 comments) Thanks for the review comments. http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/service/query-options.cc@761 PS6, Line 761: > nit: could be moved to the previous line Done http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h File be/src/util/codec.h: http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h@63 PS6, Line 63: THdfsCompression::type format_; : // Currently only ZSTD uses compression level. : int compression_level_; : }; > I would prefer to make the members public. If you want to keep private memb Made the members public. http://gerrit.cloudera.org:8080/#/c/13396/6/be/src/util/codec.h@68 PS6, Line 68: eate a > Can you mention in a comment that currently only ZSTD uses this? Done http://gerrit.cloudera.org:8080/#/c/13396/6/cmake_modules/FindZstd.cmake File cmake_modules/FindZstd.cmake: http://gerrit.cloudera.org:8080/#/c/13396/6/cmake_modules/FindZstd.cmake@36 PS6, Line 36: > nit: whitespace consistency Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py File tests/query_test/test_insert_parquet.py: http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@127 PS5, Line 127: > flake8: E302 expected 2 blank lines, found 1 Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@144 PS5, Line 144: > flake8: E115 expected an indented block (comment) Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@179 PS5, Line 179: > flake8: E703 statement ends with a semicolon Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@181 PS5, Line 181: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@181 PS5, Line 181: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@182 PS5, Line 182: t > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@182 PS5, Line 182: . > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@182 PS5, Line 182: _ > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@182 PS5, Line 182: l > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@183 PS5, Line 183: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@183 PS5, Line 183: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@183 PS5, Line 183: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@183 PS5, Line 183: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@184 PS5, Line 184: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@184 PS5, Line 184: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@185 PS5, Line 185: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@185 PS5, Line 185: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@186 PS5, Line 186: s > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@186 PS5, Line 186: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@186 PS5, Line 186: T > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@187 PS5, Line 187: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_test/test_insert_parquet.py@188 PS5, Line 188: > flake8: E231 missing whitespace after ',' Done http://gerrit.cloudera.org:8080/#/c/13396/5/tests/query_te
