[Impala-ASF-CR] IMPALA-8850: is percent not initialized in TmpFileMgr parser
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14040 ) Change subject: IMPALA-8850: is_percent not initialized in TmpFileMgr parser .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/14040/2/be/src/util/parse-util-test.cc File be/src/util/parse-util-test.cc: http://gerrit.cloudera.org:8080/#/c/14040/2/be/src/util/parse-util-test.cc@40 PS2, Line 40: is_percent = true; nit: maybe add a short comment here or as a method comment explaining why we are initializing it every time to the opposite of what is expected. -- To view, visit http://gerrit.cloudera.org:8080/14040 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4465944c7043c1fc6b33db9693ec290c24e3ed19 Gerrit-Change-Number: 14040 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 10 Aug 2019 01:22:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8848: fix UNION missing input cardinality bug
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14036 ) Change subject: IMPALA-8848: fix UNION missing input cardinality bug .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4212/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14036 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic3ed670ffb685d8ff24824933ca303f3219737bb Gerrit-Change-Number: 14036 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 10 Aug 2019 00:34:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 5: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/445/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 10 Aug 2019 00:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 7: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/444/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 10 Aug 2019 00:17:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Hello Gabor Kaszab, Attila Jeges, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13983 to look at the new patch set (#5). Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. IMPALA-7374: [DOCS] Document the new DATE data type in Impala Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 --- M docs/impala.ditamap M docs/topics/impala_date.xml M docs/topics/impala_literals.xml 3 files changed, 329 insertions(+), 189 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/13983/5 -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 5: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/445/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 10 Aug 2019 00:17:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 7: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/444/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 10 Aug 2019 00:09:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Hello Greg Rahn, Tamas Mate, Lars Volker, Zoltan Borok-Nagy, Attila Jeges, Tim Armstrong, Csaba Ringhofer, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14021 to look at the new patch set (#7). Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala - Added impala_transactions.xml - Added impala_default_transactional_type.xml Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb --- M docs/impala.ditamap M docs/topics/impala_create_table.xml A docs/topics/impala_default_transactional_type.xml A docs/topics/impala_transactions.xml 4 files changed, 292 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/14021/7 -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/14021/6/docs/topics/impala_default_transactional_type.xml File docs/topics/impala_default_transactional_type.xml: http://gerrit.cloudera.org:8080/#/c/14021/6/docs/topics/impala_default_transactional_type.xml@58 PS6, Line 58: The : INSERT operations will be transactional on this table. > I think that the best thing here would be a link to impala_transactions. Done http://gerrit.cloudera.org:8080/#/c/14021/5/docs/topics/impala_transactions.xml File docs/topics/impala_transactions.xml: http://gerrit.cloudera.org:8080/#/c/14021/5/docs/topics/impala_transactions.xml@35 PS5, Line 35: satisfy ACID (Atomicity, Consistency, Isolation, Durability) properties, which intend to > I totally agree with Tim, we should think it through carefully what to writ Rewored more. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 10 Aug 2019 00:07:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8160: [DOCS] Document CAST (...FORMAT..) function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14044 ) Change subject: IMPALA-8160: [DOCS] Document CAST (...FORMAT..) function .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/443/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6235ffd03ac56e648552058ff02491a55289c092 Gerrit-Change-Number: 14044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 23:59:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8848: fix UNION missing input cardinality bug
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14036 to look at the new patch set (#3). Change subject: IMPALA-8848: fix UNION missing input cardinality bug .. IMPALA-8848: fix UNION missing input cardinality bug If a UNION has children, and none of those children has a known cardinality, then we can make no reasonable estimate of the output cardinality, so the planner should consider the output cardinality to be unknown. The previous behaviour was to report a cardinality of 0, which is unsafe because the planner may make further decisions under the incorrect assumption that the output of the UNION is tiny. Testing: An existing CardinalityTest already tested this but had the wrong estimate. Change-Id: Ic3ed670ffb685d8ff24824933ca303f3219737bb --- M fe/src/main/java/org/apache/impala/planner/UnionNode.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java 2 files changed, 27 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/14036/3 -- To view, visit http://gerrit.cloudera.org:8080/14036 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic3ed670ffb685d8ff24824933ca303f3219737bb Gerrit-Change-Number: 14036 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8160: [DOCS] Document CAST (...FORMAT..) function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14044 ) Change subject: IMPALA-8160: [DOCS] Document CAST (...FORMAT..) function .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/443/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14044 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6235ffd03ac56e648552058ff02491a55289c092 Gerrit-Change-Number: 14044 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 23:46:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#4). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties Hive depends on property COLUMN_STATS_ACCURATE to tell if the stored statistics accurate. After Impala inserts data, it does not set statistics values up-to-date(for example numRows). Impala should unset COLUMN_STATS_ACCURATE to tell Hive the stored stats are no longer accurate. The patch impletes: After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implements the stats changes above for both acid/non-acid tables. Tests: Manual tests. Run core tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 340 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/4 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14024 ) Change subject: IMPALA-8766: Undo hadoop-cloud-storage + HWX Nexus .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh File bin/impala-config.sh: http://gerrit.cloudera.org:8080/#/c/14024/2/bin/impala-config.sh@219 PS2, Line 219: UNUSED Why do we need this env variable in case of hive-2? May be add a comment since its not obvious to me. -- To view, visit http://gerrit.cloudera.org:8080/14024 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I79a0c2575fc50bbc3b393c150c0bce22258ea1bd Gerrit-Change-Number: 14024 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 09 Aug 2019 21:20:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14038 ) Change subject: IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables .. Patch Set 5: (3 comments) http://gerrit.cloudera.org:8080/#/c/14038/5/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/14038/5/fe/src/main/java/org/apache/impala/service/Frontend.java@510 PS5, Line 510: table Can this table be instanceof Incomplete table in case there is a tableloading exception during analysis? http://gerrit.cloudera.org:8080/#/c/14038/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1721 PS5, Line 1721: transactionKeepalive_.deleteLock(lockId); : MetastoreShim.releaseLock(client.getHiveClient(), lockId); Is it possible that we are silently leaking locks if for instance MetastoreShim.releaseLock throws an exception (say due to connection issue with HMS)? According to me both these statements should happen atomically. Either they both succeed or not. http://gerrit.cloudera.org:8080/#/c/14038/5/fe/src/main/java/org/apache/impala/service/Frontend.java@1809 PS5, Line 1809: lockId = MetastoreShim.acquireLock(client.getHiveClient(), 0L, lockComponents, : LOCK_RETRIES, LOCK_RETRY_WAIT_SECONDS); : transactionKeepalive_.addLock(lockId, queryCtx); same comment as previous regarding the atomicity of these two statemnts. May be provide this as a API in the transactionKeepAlive so that we can reuse that code elsewhere if needed. -- To view, visit http://gerrit.cloudera.org:8080/14038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 Gerrit-Change-Number: 14038 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 09 Aug 2019 21:14:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14021 ) Change subject: IMPALA-8811 IMPALA-8813: [DOCS] Document the ACID support in Impala .. Patch Set 6: (2 comments) http://gerrit.cloudera.org:8080/#/c/14021/6/docs/topics/impala_default_transactional_type.xml File docs/topics/impala_default_transactional_type.xml: http://gerrit.cloudera.org:8080/#/c/14021/6/docs/topics/impala_default_transactional_type.xml@58 PS6, Line 58: The : INSERT operations will be transactional on this table. I think that the best thing here would be a link to impala_transactions. http://gerrit.cloudera.org:8080/#/c/14021/5/docs/topics/impala_transactions.xml File docs/topics/impala_transactions.xml: http://gerrit.cloudera.org:8080/#/c/14021/5/docs/topics/impala_transactions.xml@35 PS5, Line 35: satisfy a level of ACID (Atomicity, Consistency, Isolation, Durability) properties, which > Slightly reworded. I totally agree with Tim, we should think it through carefully what to write here. What Impala should profit from Hive ACID at the moment is that a single table insert from Hive/Impala is either completely seen or not seen at all during a query (without ACID if an insert create more than one files, it was possible to list the files at the "wrong moment" when only some of them were copied to their final destination from the staging directory), and for a single table the inserts are "ordered", so if Impala doesn't see a committed insert, it won't see any insert committed after it. On the other side for different tables Impala can have snapshots from totally different times, so it may see a commit on table A while not see a much earlier commit on table B. This should not happen when reading with Hive. -- To view, visit http://gerrit.cloudera.org:8080/14021 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0320e573301175dedb00d6e745f9f973519a16bb Gerrit-Change-Number: 14021 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 09 Aug 2019 20:44:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 3: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4766/ -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Fri, 09 Aug 2019 20:41:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 09 Aug 2019 17:54:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7322: Add storage wait time to profile
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13786 ) Change subject: IMPALA-7322: Add storage wait time to profile .. Patch Set 1: Can you rebase? I can submit for a GVO. -- To view, visit http://gerrit.cloudera.org:8080/13786 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7447f8c8e7e50eb71d18643859d2e3de865368d2 Gerrit-Change-Number: 13786 Gerrit-PatchSet: 1 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Fri, 09 Aug 2019 17:14:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/14012 )
Change subject: IMPALA-4551: Limit the size of SQL statements
..
Patch Set 1:
(4 comments)
lgtm overall. Some general comments.
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@462
PS1, Line 462: analysisResult_.analyzer_.checkStmtExprLimit();
nit: Probably worth mentioning that we want to enforce this before the rewrites
because rewrites are costly with long expression chains.
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java:
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@2435
PS1, Line 2435: String repCols20 = getRepeatedColumnReference("int_col",
20, true);
Verify that analyzer.numStmtExprs_ is accounted properly?
http://gerrit.cloudera.org:8080/#/c/14012/1/fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java@2493
PS1, Line 2493: StringBuilder inList = new StringBuilder();
what about constant expressions in the IN lists? foo IN (2*3, 3*4...) ? It
looks to me like they are accounted, add tests for a mix of Literal and Const
expressions?
http://gerrit.cloudera.org:8080/#/c/14012/1/tests/query_test/test_exprs.py
File tests/query_test/test_exprs.py:
http://gerrit.cloudera.org:8080/#/c/14012/1/tests/query_test/test_exprs.py@139
PS1, Line 139: # This takes 20+ minutes, so only run it on exhaustive.
Is the intention to test the default limits here? If not, we can probably set a
non-default lower expression limit to save time and memory.
--
To view, visit http://gerrit.cloudera.org:8080/14012
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c
Gerrit-Change-Number: 14012
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Quanlong Huang
Gerrit-Comment-Date: Fri, 09 Aug 2019 17:11:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14039 to look at the new patch set (#3). Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. IMPALA-8818: Replace deque with spillable queue in BufferedPRS Replaces DequeRowBatchQueue with SpillableRowBatchQueue in BufferedPlanRootSink. A few changes to BufferedPlanRootSink were necessary for it to work with the spillable queue, however, all the synchronization logic is the same. SpillableRowBatchQueue is a wrapper around a BufferedTupleStream and a ReservationManager. It takes in a TBackendResourceProfile that specifies the max / min memory reservation the BufferedTupleStream can use to buffer rows. The 'max_unpinned_bytes' parameter limits the max number of bytes that can be unpinned in the BufferedTupleStream. The limit is a 'soft' limit because calls to AddBatch may push the amount of unpinned memory over the limit. The queue is non-blocking and not thread safe. It provides AddBatch and GetBatch methods. Calls to AddBatch spill if the BufferedTupleStream does not have enough reservation to fit the entire RowBatch. Adds two new query options: 'MAX_PINNED_RESULT_SPOOLING_MEMORY' and 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY', which bound the amount of pinned and unpinned memory that a query can use for spooling, respectively. MAX_PINNED_RESULT_SPOOLING_MEMORY must be <= MAX_UNPINNED_RESULT_SPOOLING_MEMORY in order to allow all the pinned data in the BufferedTupleStream to be unpinned. This is enforced in a new method in QueryOptions called 'ValidateQueryOptions'. Planner Changes: PlanRootSink.java now computes a full ResourceProfile if result spooling is enabled. The min mem reservation is bounded by the size of the read and write pages used by the BufferedTupleStream. The max mem reservation is bounded by 'MAX_PINNED_RESULT_SPOOLING_MEMORY'. The mem estimate is computed by estimating the size of the result set using stats. BufferedTupleStream Re-Factoring: For the most part, using a BufferedTupleStream outside an ExecNode works properly. However, some changes were necessary: * The message for the MAX_ROW_SIZE error is ExecNode specific. In order to fix this, this patch introduces the concept of an ExecNode 'label' which is a more generic version of an ExecNode 'id'. * The definition of TBackendResourceProfile lived in PlanNodes.thrift, it was moved to its own file so it can be used by DataSinks.thrift. * Modified BufferedTupleStream so it internally tracks how many bytes are unpinned (necessary for 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY'). Metrics: * Added a few of the metrics mentioned in IMPALA-8825 to BufferedPlanRootSink. Specifically, added timers to track how much time is spent waiting in the BufferedPlanRootSink 'Send' and 'GetNext' methods. * The BufferedTupleStream in the SpillableRowBatchQueue exposes several BufferPool metrics such as number of reserved and unpinned bytes. Bug Fixes: * Fixed a bug in BufferedPlanRootSink where the MemPool used by the expression evaluators was not being cleared incrementally. * Fixed a bug where the inactive timer was not being properly updated in BufferedPlanRootSink. * Fixed a bug where RowBatch memory was not freed if BufferedPlanRootSink::GetNext terminated early because it could not handle requests where num_results < BATCH_SIZE. Testing: * Added new tests to test_result_spooling.py. * Updated errors thrown in spilling-large-rows.test. * Ran core tests. Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/analytic-eval-node.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/data-sink.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/plan-root-sink.h M be/src/exec/sort-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h A be/src/runtime/spillable-row-batch-queue.cc A be/src/runtime/spillable-row-batch-queue.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CMakeLists.txt M common/thrift/DataSinks.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift A
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 3: > Uploaded patch set 3. Fixed a few tests failures. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 09 Aug 2019 17:12:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Fri, 09 Aug 2019 17:09:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14012 ) Change subject: IMPALA-4551: Limit the size of SQL statements .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4208/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14012 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c Gerrit-Change-Number: 14012 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 09 Aug 2019 16:52:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8755: Frontend support for Z-ordering
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13955 ) Change subject: IMPALA-8755: Frontend support for Z-ordering .. Patch Set 6: (4 comments) http://gerrit.cloudera.org:8080/#/c/13955/6/common/thrift/DataSinks.thrift File common/thrift/DataSinks.thrift: http://gerrit.cloudera.org:8080/#/c/13955/6/common/thrift/DataSinks.thrift@83 PS6, Line 83: // Sorting algorithm. If not lexical, the backend should not populate the : // RowGroup::sorting_columns list in parquet files. : 7: required Types.TSortingAlgorithm sorting_algorithm I think it's rather an 'order' than an 'algorithm'. Algorithm is e.g. quicksort that sorts your data into some specified order. Since you use it quite excessively I think the easiest way to search and replace it is to download the patch from gerrit, edit the patch file and replace 'algorithm' to 'order' and then apply the patch to a new branch. nit: I think you can leave out stuff about Parquet because it's just some low-level detail. http://gerrit.cloudera.org:8080/#/c/13955/6/fe/src/main/cup/sql-parser.cup File fe/src/main/cup/sql-parser.cup: http://gerrit.cloudera.org:8080/#/c/13955/6/fe/src/main/cup/sql-parser.cup@1399 PS6, Line 1399: RESULT = new CreateTableLikeStmt(tbl_def.getTblName(), nit: whitespaces at line end http://gerrit.cloudera.org:8080/#/c/13955/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java: http://gerrit.cloudera.org:8080/#/c/13955/6/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2570 PS6, Line 2570: nit: missing words http://gerrit.cloudera.org:8080/#/c/13955/6/testdata/workloads/functional-query/queries/QueryTest/create-table-like-table-zorder.test File testdata/workloads/functional-query/queries/QueryTest/create-table-like-table-zorder.test: http://gerrit.cloudera.org:8080/#/c/13955/6/testdata/workloads/functional-query/queries/QueryTest/create-table-like-table-zorder.test@24 PS6, Line 24: string_col We shouldn't allow string cols to be sorted by ZORDER. It's better to prohibit it during analysis. -- To view, visit http://gerrit.cloudera.org:8080/13955 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie122002ca8f52ca2c1e1ec8ff1d476ae1f4f875d Gerrit-Change-Number: 13955 Gerrit-PatchSet: 6 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 09 Aug 2019 16:46:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14043 ) Change subject: WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4207/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 Gerrit-Change-Number: 14043 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 16:37:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 16:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14026 ) Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4766/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho Gerrit-Comment-Date: Fri, 09 Aug 2019 16:28:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters
Hello Michael Ho, Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14026 to look at the new patch set (#3). Change subject: IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters .. IMPALA-8685,IMPALA-8677: Use consistent scheduling for small clusters In the original change for consistent scheduling, if a cluster has fewer nodes than the number of remote executor candidates, then the scheduler falls back to using the old SelectRemoteExecutor(). SelectRemoteExecutor() considers all backends and picks the backend with the least assigned bytes; to break ties, it uses randomness. This means that clusters with fewer backends than num_remote_executor_candidates do not have consistent placement. For the file handle cache (the original user of consistent placement), this is not a major problem. However, for data caching, it can result in slower warm up of the data cache and greater duplication of the same data across different nodes. This changes the algorithm to use consistent placement even for small clusters (num nodes <= num_remote_executor_candidates). To make this more predictable, it increases the maximum number of iterations. This also changes GetRemoteExecutorCandidates() to return the candidates in the order that they were selected. In code, this means looping over a vector to detect distinct backends rather than using a set. Testing: - Modify the scheduler-test backend test to verify that small clusters use consistent scheduling. Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 --- M be/src/scheduling/scheduler-test.cc M be/src/scheduling/scheduler.cc 2 files changed, 75 insertions(+), 44 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/14026/3 -- To view, visit http://gerrit.cloudera.org:8080/14026 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Icfdb2cc53d7206e316ea8a1cc28ad443f246f741 Gerrit-Change-Number: 14026 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8848: fix UNION missing input cardinality bug
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/14036 ) Change subject: IMPALA-8848: fix UNION missing input cardinality bug .. Patch Set 2: (4 comments) lgtm, just a bunch of clarifying questions and nits. http://gerrit.cloudera.org:8080/#/c/14036/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14036/2//COMMIT_MSG@9 PS2, Line 9: If a UNION has input of unknown size, we should nit: I got confused by this statement for a minute. Rephrase may be? (to clarify that atleast one child with a valid cardinality is needed) http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java File fe/src/main/java/org/apache/impala/planner/UnionNode.java: http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java@124 PS2, Line 124: nit:..atleast..? http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/main/java/org/apache/impala/planner/UnionNode.java@127 PS2, Line 127: cardinality_ = checkedAdd(totalChildCardinality, constExprLists_.size()); nit:Preconditions.check(constExprLists_.size() > 0)? http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java: http://gerrit.cloudera.org:8080/#/c/14036/2/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@724 PS2, Line 724: path, UnionNode.class); Should we also add a test to cover something like select * from tbl_has_valid_cards union select * from tbl_without_valid_cards ? -- To view, visit http://gerrit.cloudera.org:8080/14036 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic3ed670ffb685d8ff24824933ca303f3219737bb Gerrit-Change-Number: 14036 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 16:25:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14012 ) Change subject: IMPALA-4551: Limit the size of SQL statements .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4765/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14012 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c Gerrit-Change-Number: 14012 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 09 Aug 2019 16:12:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14012 ) Change subject: IMPALA-4551: Limit the size of SQL statements .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/14012/2/tests/query_test/test_exprs.py File tests/query_test/test_exprs.py: http://gerrit.cloudera.org:8080/#/c/14012/2/tests/query_test/test_exprs.py@216 PS2, Line 216: flake8: E261 at least two spaces before inline comment -- To view, visit http://gerrit.cloudera.org:8080/14012 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c Gerrit-Change-Number: 14012 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Fri, 09 Aug 2019 16:12:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4551: Limit the size of SQL statements
Hello Bharath Vissapragada, Michael Ho, Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14012 to look at the new patch set (#2). Change subject: IMPALA-4551: Limit the size of SQL statements .. IMPALA-4551: Limit the size of SQL statements Various BI tools generate and run SQL. When used incorrectly or misconfigured, the tools can generate extremely large SQLs. Some of these SQL statements reach 10s of megabytes. Large SQL statements impose costs throughout execution, including statement rewrite logic in the frontend and codegen in the backend. The resource usage of these statements can impact the stability of the system or the ability to run other SQL statements. This implements two new query options that provide controls to reject large SQL statements. - The first, MAX_STATEMENT_LENGTH_BYTES is a cap on the total size of the SQL statement (in bytes). It is applied before any parsing or analysis. It uses a default value of 16MB. - The second, STATEMENT_EXPRESSION_LIMIT, is a limit on the total number of expressions in a statement or any views that it references. The limit is applied upon the first round of analysis, but it is not reapplied when statement rewrite rules are applied. Certain expressions such as literals in IN lists or VALUES clauses are not analyzed and do not count towards the limit. It uses a default value of 250,000. The two are complementary. Since enforcing the statement expression limit requires parsing and analyzing the statement, the MAX_STATEMENT_LENGTH_BYTES sets an upper bound on the size of statement that needs to be parsed and analyzed. Testing confirms that even statements approaching 16MB get through the first round of analysis within a few seconds and then are rejected. Testing: - This adds frontend tests that verify the low level semantics about how expressions are counted and verifies that the expression limits are enforced. - This adds end-to-end tests that verify both the MAX_STATEMENT_LENGTH_BYTES and STATEMENT_EXPRESSION_LIMIT at their defaults values. - There is also an end-to-end test that runs in exhaustive mode that runs a SQL with close to 250,000 expressions. Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c --- M be/src/service/impala-server.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/generate_error_codes.py M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M tests/query_test/test_exprs.py 12 files changed, 277 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/14012/2 -- To view, visit http://gerrit.cloudera.org:8080/14012 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I5675fb4a08c1dc51ae5bcf467cbb969cc064602c Gerrit-Change-Number: 14012 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2.
Attila Jeges has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14043 ) Change subject: WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. .. WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. This change is Step 2 in a series of steps to coordinate the introduction of HMS API changes to Hive and Impala: 1. Change in Hive. We push new APIs so Impala can use them. New APIs will simply call old existing methods so there should not be any change of functionality there. Since there were many incompatible changes, new APIs are tagged method_name_V2. 2. Change in Impala. Push changes to use new methods *V2. 3. Change in Hive Push patch with complete functionality. *V2 methods contains the new logic. The old existing methods are not used anymore by Impala at this point, hence they can be removed. For every method_name_V2, I will create a corresponding method method_name that calls the former one. 4. Change in Impala Replace *V2 calls by *. 5. Change in Hive. Remove *V2 methods from API. This patch-set is in WIP state as Step 1 is not yet complete. TODO: Once Step 1 is complete, make the new build available to Impala via the native-toolchain. Update the CDP_BUILD_NUMBER and CDP_HIVE_VERSION environment variables in bin/impala-config.sh accordingly. Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 6 files changed, 90 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14043/2 -- To view, visit http://gerrit.cloudera.org:8080/14043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 Gerrit-Change-Number: 14043 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 7: (6 comments) http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/12443/7/be/src/service/impala-http-handler.cc@109 PS7, Line 109: // The /catalog_object endpoint is disabled if local_catalog_mode is used. nit: add why http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@605 PS7, Line 605: execute_serially why? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@607 PS7, Line 607: A positive test for /catalog_ob Instead say, check /catalog_obj endpoint is not accessible or something? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@608 PS7, Line 608: http://localhost:25000 use the URL templates above? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@609 PS7, Line 609: assert 'No URI handler for /catalog_object' \ I'm confused, the description says something else? *not* in catalog mode? http://gerrit.cloudera.org:8080/#/c/12443/7/tests/webserver/test_web_pages.py@610 PS7, Line 610:not in response Can you also scrape the webpage source and make sure it doesn't have a "catalog_object" anywhere? -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 7 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 15:54:24 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14043 ) Change subject: WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4206/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 Gerrit-Change-Number: 14043 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 15:37:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14037 ) Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4205/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 15:19:59 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2.
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14043 Change subject: WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. .. WIP: IMPALA-8842: Accommodate HIVE-22046 - Step 2. HIVE-22046 added 'engine' column to TAB_COL_STATS and PART_COL_STATS HMS tables. The new column is used to differentiate among column stats computed by different engines. The related HMS API calls were changed accordingly. This change is Step 2 in a series of steps to coordinate the introduction of HMS API changes to Hive and Impala: 1. Change in Hive. We push new APIs so Impala can use them. New APIs will simply call old existing methods so there should not be any change of functionality there. Since there were many incompatible changes, new APIs are tagged method_name_V2. 2. Change in Impala. Push changes to use new methods *V2. 3. Change in Hive Push patch with complete functionality. *V2 methods contains the new logic. The old existing methods are not used anymore by Impala at this point, hence they can be removed. For every method_name_V2, I will create a corresponding method method_name that calls the former one. 4. Change in Impala Replace *V2 calls by *. 5. Change in Hive. Remove *V2 methods from API. This patch-set is in WIP state as Step 1 is not yet complete. TODO: Once Step 1 is complete, make the new build available to Impala via the native-toolchain. Update the CDP_BUILD_NUMBER and CDP_HIVE_VERSION environment variables in bin/impala-config.sh accordingly. Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java 6 files changed, 90 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/14043/1 -- To view, visit http://gerrit.cloudera.org:8080/14043 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9a73f5eeac8e84d63b22aaed5dfbcd8ea39f0af4 Gerrit-Change-Number: 14043 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-8839: Remove COLUMN STATS ACCURATE from properties
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14037 to look at the new patch set (#3). Change subject: IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties .. IMPALA-8839: Remove COLUMN_STATS_ACCURATE from properties After Impala insert data, Remove COLUMN_STATS_ACCURATE from table properties if it exists Remove COLUMN_STATS_ACCURATE from partition params if it exists Add helper methods to handle alter table/partition for acid tables. Implemented the stats change for both acid/non-acid tables. Tests: Manual tests. Add ee tests to test interop with Hive for acid/external tables. Change-Id: I13f4a77022a7112e10a07314359f927eae083deb --- M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java A testdata/workloads/functional-query/queries/QueryTest/acid-clear-statsaccurate.test A testdata/workloads/functional-query/queries/QueryTest/clear-statsaccurate.test M tests/query_test/test_acid.py 6 files changed, 338 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/14037/3 -- To view, visit http://gerrit.cloudera.org:8080/14037 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I13f4a77022a7112e10a07314359f927eae083deb Gerrit-Change-Number: 14037 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046.
Attila Jeges has abandoned this change. ( http://gerrit.cloudera.org:8080/14032 ) Change subject: WIP: IMPALA-8842: Change column stat HMS calls to accommodate HIVE-22046. .. Abandoned Accommodating HIVE-22046 in Impala will be done in multiple steps in a staged manner. -- To view, visit http://gerrit.cloudera.org:8080/14032 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: Ia782bda3891dee1281cfcd5cf8817aa1226217ad Gerrit-Change-Number: 14032 Gerrit-PatchSet: 4 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8846: Undefined behaviour in RleEncoder::Put
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14042 ) Change subject: IMPALA-8846: Undefined behaviour in RleEncoder::Put .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4204/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I269443d1f1680e672fde7dd88eab5fcb56c65613 Gerrit-Change-Number: 14042 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 13:04:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8846: Undefined behaviour in RleEncoder::Put
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14042 ) Change subject: IMPALA-8846: Undefined behaviour in RleEncoder::Put .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I269443d1f1680e672fde7dd88eab5fcb56c65613 Gerrit-Change-Number: 14042 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:42:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14038 ) Change subject: IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4203/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 Gerrit-Change-Number: 14038 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:34:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8846: Undefined behaviour in RleEncoder::Put
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14042 Change subject: IMPALA-8846: Undefined behaviour in RleEncoder::Put .. IMPALA-8846: Undefined behaviour in RleEncoder::Put To test for overflow, we used 'repeat_count_ <= std::numeric_limits::max()', but this is always true as repeat_count_ is an int. This could have lead to undefined behaviour because we increment repeat_count_ afterwards. Changed the comparison not to allow equality. Change-Id: I269443d1f1680e672fde7dd88eab5fcb56c65613 --- M be/src/util/rle-encoding.h 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/14042/1 -- To view, visit http://gerrit.cloudera.org:8080/14042 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I269443d1f1680e672fde7dd88eab5fcb56c65613 Gerrit-Change-Number: 14042 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4764/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:24:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:24:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Restrict bit unpacking to unsigned integer types
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14031 ) Change subject: IMPALA-8796: Restrict bit unpacking to unsigned integer types .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14031 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cb57f43cc93e5dda6b6b2ff1617f1631f8cbe1 Gerrit-Change-Number: 14031 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:23:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 12:17:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14038 to look at the new patch set (#5). Change subject: IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables .. IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M common/thrift/Frontend.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-query/queries/QueryTest/acid.test M tests/query_test/test_acid.py 14 files changed, 157 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/14038/5 -- To view, visit http://gerrit.cloudera.org:8080/14038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 Gerrit-Change-Number: 14038 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14038 ) Change subject: IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables .. Patch Set 4: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4202/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 Gerrit-Change-Number: 14038 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 10:21:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4201/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 10:06:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14038 to look at the new patch set (#4). Change subject: IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables .. IMPALA-8823 WIP: DROP TABLE support for insert-only ACID tables Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/service/impala-server.cc M common/thrift/Frontend.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/analysis/AnalyzerTest.java M testdata/workloads/functional-query/queries/QueryTest/acid.test 13 files changed, 152 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/14038/4 -- To view, visit http://gerrit.cloudera.org:8080/14038 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic41ca73268c4b75af5a08fe3dd1ada1df3f6fd34 Gerrit-Change-Number: 14038 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4200/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 09:58:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Daniel Becker has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. IMPALA-8796: Add unit tests to UnpackAndDecodeValues Adding unit tests to test BitPacking::UnpackAndDecodeValues. Also changing the existing BitPacking tests so that they are parametrised by the output type of the unpacking so we can test different output types. Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 --- M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.cc 2 files changed, 195 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14004/5 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 5 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8796: Add unit tests to UnpackAndDecodeValues
Daniel Becker has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/14004 ) Change subject: IMPALA-8796: Add unit tests to UnpackAndDecodeValues .. IMPALA-8796: Add unit tests to UnpackAndDecodeValues Adding unit tests to test BitPacking::UnpackAndDecodeValues. Also changing the existing BitPacking tests so that they are parametrised by the output type of the unpacking so we can test different output types. Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 --- M be/src/util/bit-packing-test.cc M be/src/util/bit-packing.cc 2 files changed, 195 insertions(+), 25 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14004/4 -- To view, visit http://gerrit.cloudera.org:8080/14004 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I363ff9f5d4750231fdc1aa60cf40d90ebb9ba305 Gerrit-Change-Number: 14004 Gerrit-PatchSet: 4 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-7374: [DOCS] Document the new DATE data type in Impala
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/13983 ) Change subject: IMPALA-7374: [DOCS] Document the new DATE data type in Impala .. Patch Set 4: Code-Review+1 I'm fine with the review. Let's wait for Attila Jeges to take another look. -- To view, visit http://gerrit.cloudera.org:8080/13983 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0c28361c7f0d225708eaf4b955c6704520eaaa68 Gerrit-Change-Number: 13983 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 09 Aug 2019 07:27:38 + Gerrit-HasComments: No
