[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Attila Jeges has removed Tim Armstrong from this change. ( http://gerrit.cloudera.org:8080/14852 ) Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. Removed reviewer Tim Armstrong. -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Update gitignore files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 04:38:08 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Update gitignore files This adds in a handful of files that I had on my local machine Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Reviewed-on: http://gerrit.cloudera.org:8080/14858 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M .gitignore M fe/.gitignore 2 files changed, 16 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14813 ) Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5220/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Dec 2019 03:05:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14813 ) Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5219/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 2 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Dec 2019 02:56:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/14813 ) Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins .. Patch Set 3: Quanlong, I have uploaded additional changes after addressing review comments. I have kept the original fix plus fixed the values transfer issue you found. Could you pls take another look ? Thanks. -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Dec 2019 02:40:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Hello Quanlong Huang, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14813 to look at the new patch set (#3). Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins .. IMPALA-9162: Do not apply inferred predicate to outer joins When the planner migrates predicates to inline views, it also creates equivalent predicates based on the value transfer graph which is built by transitive relationships among join conditions. These newly inferred predicates are placed typically as 'other predicates' of an inner or outer join. However, for outer joins, this has the effect of adding extra predicates in the WHERE clause which is incorrect since it may filter NULL values. Since the original query did not have null filtering conditions in the WHERE clause, we should not add new ones. This fix does 2 things: - At the time of creating the values transfer graph in Analyzer, don't add a directed edge between two slots if the first slot references the output tuple of an outer join. - During the migration of conjuncts to inline views, analyze the predicate of type A B and if it is an inferred predicate AND either the left or right slots reference the output tuple of an outer join, the inferred predicate is ignored. This serves as a safety check in case any unqualified predicate 'fell through' until this stage of planning. Note that simple queries with combination of inner and outer joins may not reproduce the problem. Due to the nature of predicate inferencing, some combination of subqueries, inner joins, outer joins is needed. For the query pattern, please see the example in the JIRA. Tests: - Added plan tests with left and right outer joins to inline-view.test - Manually ran few queries on impala shell to verify result correctness: by checking that NULL values are being produced for outer joins. - Ran regression tests on jenkins Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test 4 files changed, 172 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/14813/3 -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 3 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/14813 )
Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2238
PS2, Line 2238: LOG.trace("Considering value transfer between " +
slotRefs.first.getSlotId().toString() +
line too long (99 > 90)
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2244
PS2, Line 2244: g.addEdge(slotRefs.first.getSlotId().asInt(),
slotRefs.second.getSlotId().asInt());
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2246
PS2, Line 2246: LOG.trace("value transfer: from " +
slotRefs.first.getSlotId().toString() + " to " +
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2253
PS2, Line 2253: g.addEdge(slotRefs.second.getSlotId().asInt(),
slotRefs.first.getSlotId().asInt());
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/14813/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2255
PS2, Line 2255: LOG.trace("value transfer: from " +
slotRefs.second.getSlotId().toString() + " to " +
line too long (97 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/14813
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827
Gerrit-Change-Number: 14813
Gerrit-PatchSet: 2
Gerrit-Owner: Aman Sinha
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 06 Dec 2019 02:27:08 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9162: Do not apply inferred predicate to outer joins
Hello Quanlong Huang, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14813 to look at the new patch set (#2). Change subject: IMPALA-9162: Do not apply inferred predicate to outer joins .. IMPALA-9162: Do not apply inferred predicate to outer joins When the planner migrates predicates to inline views, it also creates equivalent predicates based on the value transfer graph which is built by transitive relationships among join conditions. These newly inferred predicates are placed typically as 'other predicates' of an inner or outer join. However, for outer joins, this has the effect of adding extra predicates in the WHERE clause which is incorrect since it may filter NULL values. Since the original query did not have null filtering conditions in the WHERE clause, we should not add new ones. This fix does 2 things: - At the time of creating the values transfer graph in Analyzer, don't add a directed edge between two slots if the first slot references the output tuple of an outer join. - During the migration of conjuncts to inline views, analyze the predicate of type A B and if it is an inferred predicate AND either the left or right slots reference the output tuple of an outer join, the inferred predicate is ignored. This serves as a safety check in case any unqualified predicate 'fell through' until this stage of planning. Note that simple queries with combination of inner and outer joins may not reproduce the problem. Due to the nature of predicate inferencing, some combination of subqueries, inner joins, outer joins is needed. For the query pattern, please see the example in the JIRA. Tests: - Added plan tests with left and right outer joins to inline-view.test - Manually ran few queries on impala shell to verify result correctness: by checking that NULL values are being produced for outer joins. - Ran regression tests on jenkins Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M testdata/workloads/functional-planner/queries/PlannerTest/inline-view.test 4 files changed, 169 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/14813/2 -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 2 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 10: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 06 Dec 2019 01:28:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. IMPALA-9126: part 4: hash join builder manages spilling This is the final patch for IMPALA-9126. This will allow the many:1 relationship of probe:build partitions that we need for the shared join build. Key changes: * Builder picks the next spilled partition to process. * Partitions are identified by unique ID so can be decoupled between build and probe. * unique_ptr is used to manage build partitions. This helps document the lifecycle of the partitions better, particularly when they are handed off to PartitionedHashJoinNode. Testing: * Ran exhaustive tests. * Ran a single node TPC-H and TPC-DS stress test with 1000 queries. Perf: Ran a single node TPC-H 30 test against master from before IMPALA-9126 changes. No significant perf change. Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Reviewed-on: http://gerrit.cloudera.org:8080/14790 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h 6 files changed, 182 insertions(+), 120 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 11 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14861 ) Change subject: IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3 .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/547/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0586445c13cc6d02548be4cca43014066e0d99b Gerrit-Change-Number: 14861 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 00:29:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5318/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 00:12:31 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 00:12:30 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Patch Set 1: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 00:09:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14861 Change subject: IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3 .. IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3 The following patters and documented: MONTH, MON, DAY, DY, D, Q, WW, W Also added boundaries for TZM and TZM (IMPALKA-9217). Change-Id: If0586445c13cc6d02548be4cca43014066e0d99b --- M docs/topics/impala_conversion_functions.xml 1 file changed, 705 insertions(+), 303 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/14861/1 -- To view, visit http://gerrit.cloudera.org:8080/14861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If0586445c13cc6d02548be4cca43014066e0d99b Gerrit-Change-Number: 14861 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14861 ) Change subject: IMPALA-9219: [DOCS] ISO-SQL 2016 Date/time format patters - Milestone 3 .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/547/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If0586445c13cc6d02548be4cca43014066e0d99b Gerrit-Change-Number: 14861 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 06 Dec 2019 00:08:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9207: [DOCS] Documented the #Inst in exec summary
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14860 ) Change subject: IMPALA-9207: [DOCS] Documented the #Inst in exec summary .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/546/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I938930c66144ba6bce766981d363abe4b28ba524 Gerrit-Change-Number: 14860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 23:14:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9207: [DOCS] Documented the #Inst in exec summary
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14860 ) Change subject: IMPALA-9207: [DOCS] Documented the #Inst in exec summary .. Patch Set 1: The actual change starts at line 147 through 150. -- To view, visit http://gerrit.cloudera.org:8080/14860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I938930c66144ba6bce766981d363abe4b28ba524 Gerrit-Change-Number: 14860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 22:52:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9207: [DOCS] Documented the #Inst in exec summary
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14860 Change subject: IMPALA-9207: [DOCS] Documented the #Inst in exec summary .. IMPALA-9207: [DOCS] Documented the #Inst in exec summary Change-Id: I938930c66144ba6bce766981d363abe4b28ba524 --- M docs/topics/impala_explain_plan.xml 1 file changed, 153 insertions(+), 481 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/14860/1 -- To view, visit http://gerrit.cloudera.org:8080/14860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I938930c66144ba6bce766981d363abe4b28ba524 Gerrit-Change-Number: 14860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-9207: [DOCS] Documented the #Inst in exec summary
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14860 ) Change subject: IMPALA-9207: [DOCS] Documented the #Inst in exec summary .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/546/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14860 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I938930c66144ba6bce766981d363abe4b28ba524 Gerrit-Change-Number: 14860 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 22:51:17 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14858 ) Change subject: Update gitignore files .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5218/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 22:17:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14811 ) Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5316/ -- To view, visit http://gerrit.cloudera.org:8080/14811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d Gerrit-Change-Number: 14811 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 22:01:22 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update impala proxy.xml with the latest info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14857 ) Change subject: [DOCS] Update impala_proxy.xml with the latest info .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/545/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9d80e21abb385704eea863d221e333441af9a39 Gerrit-Change-Number: 14857 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vincent Tran Gerrit-Comment-Date: Thu, 05 Dec 2019 21:52:26 + Gerrit-HasComments: No
[Impala-ASF-CR] Update gitignore files
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14858 Change subject: Update gitignore files .. Update gitignore files This adds in a handful of files that I had on my local machine Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd --- M .gitignore M fe/.gitignore 2 files changed, 16 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/58/14858/1 -- To view, visit http://gerrit.cloudera.org:8080/14858 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I357441fab00ac031fbc70c40e4574e7a723fdedd Gerrit-Change-Number: 14858 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14764 ) Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5217/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:42:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14764 ) Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. Patch Set 9: Code-Review+1 (6 comments) A couple of very minor comments. I can upgrade to a +2 once other reviewers have had a chance to finish their reviews. http://gerrit.cloudera.org:8080/#/c/14764/5/be/src/exec/exec-node.h File be/src/exec/exec-node.h: http://gerrit.cloudera.org:8080/#/c/14764/5/be/src/exec/exec-node.h@66 PS5, Line 66: public: > will update the comment. I initially started with only moving the Exprs out I'm fine if you stage the work in the way you think will be most efficient, I don't want to mess with your process, just understand the next steps. http://gerrit.cloudera.org:8080/#/c/14764/5/be/src/exec/exec-node.h@83 PS5, Line 83: > Eventually all should be accessible by ExecNode. In that case would you rec Maybe make it a friend class of ExecNode? And leave a TODO to clean that up in a later patch. http://gerrit.cloudera.org:8080/#/c/14764/5/be/src/exec/exec-node.h@358 PS5, Line 358: return reservation_manager_.ReleaseUnusedReservation(); > this is returning a reference now so dynamic_cast on it will actually throw Oh I didn't know that subtlety about dynamic_cast. This seems fine then as an intermediate step. http://gerrit.cloudera.org:8080/#/c/14764/5/be/src/exec/exec-node.h@382 PS5, Line 382: > yup, would unfortunately result in a cascade of changes ok, no prob. Leave a TODO maybe just so it's clear that it's not intentionally mutable. http://gerrit.cloudera.org:8080/#/c/14764/9/be/src/exec/hdfs-scan-node-base.cc File be/src/exec/hdfs-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/14764/9/be/src/exec/hdfs-scan-node-base.cc@89 PS9, Line 89: const TTupleId& tuple_id =tnode.hdfs_scan_node.tuple_id; nit: missing space http://gerrit.cloudera.org:8080/#/c/14764/9/be/src/exec/scan-node.cc File be/src/exec/scan-node.cc: http://gerrit.cloudera.org:8080/#/c/14764/9/be/src/exec/scan-node.cc@124 PS9, Line 124: // TODO: add the name of the node type This doesn't seem that necessary, the value should be enough to debug - maybe remove the TODO so there's less noise. -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:32:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14764 ) Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5216/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:30:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 9: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5215/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:29:02 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update impala proxy.xml with the latest info
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14857 Change subject: [DOCS] Update impala_proxy.xml with the latest info .. [DOCS] Update impala_proxy.xml with the latest info Change-Id: Ia9d80e21abb385704eea863d221e333441af9a39 --- M docs/topics/impala_jdbc.xml M docs/topics/impala_proxy.xml 2 files changed, 240 insertions(+), 344 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14857/1 -- To view, visit http://gerrit.cloudera.org:8080/14857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia9d80e21abb385704eea863d221e333441af9a39 Gerrit-Change-Number: 14857 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] Update impala proxy.xml with the latest info
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14857 ) Change subject: [DOCS] Update impala_proxy.xml with the latest info .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/545/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14857 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9d80e21abb385704eea863d221e333441af9a39 Gerrit-Change-Number: 14857 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 21:27:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Hello Michael Ho, Daniel Becker, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14764 to look at the new patch set (#9). Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. IMPALA-4192: Move static state from ExecNode into a PlanNode This patch adds a new class called PlanNode which contains a subset of the static state of their corresponding ExecNode, of which there is one instance per fragment. ExecNode contains the runtime state and there can be up to MT_DOP instances of it per fragment. It also adds a similar class called AggregatorConfig which contains static state initialized from the thrift aggregator struct and is passed as an input to the Aggregator class's constructor. Eventually all static state including codegened function pointers would be moved to the PlanNodes. Testing: Ran exhaustive tests successfully. Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregation-node.cc M be/src/exec/aggregation-node.h M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/cardinality-check-node.cc M be/src/exec/cardinality-check-node.h M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/empty-set-node.cc M be/src/exec/empty-set-node.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node-mt.cc M be/src/exec/kudu-scan-node-mt.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/singular-row-src-node.cc M be/src/exec/singular-row-src-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/streaming-aggregation-node.cc M be/src/exec/streaming-aggregation-node.h M be/src/exec/subplan-node.cc M be/src/exec/subplan-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h 64 files changed, 1,337 insertions(+), 699 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14764/9 -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 9 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/14764 ) Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. Patch Set 8: Rebased -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:08:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4192: Move static state from ExecNode into a PlanNode
Hello Michael Ho, Daniel Becker, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14764 to look at the new patch set (#8). Change subject: IMPALA-4192: Move static state from ExecNode into a PlanNode .. IMPALA-4192: Move static state from ExecNode into a PlanNode This patch adds a new class called PlanNode which contains a subset of the static state of their corresponding ExecNode, of which there is one instance per fragment. ExecNode contains the runtime state and there can be up to MT_DOP instances of it per fragment. It also adds a similar class called AggregatorConfig which contains static state initialized from the thrift aggregator struct and is passed as an input to the Aggregator class's constructor. Eventually all static state including codegened function pointers would be moved to the PlanNodes. Testing: Ran exhaustive tests successfully. Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregation-node.cc M be/src/exec/aggregation-node.h M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/cardinality-check-node.cc M be/src/exec/cardinality-check-node.h M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/empty-set-node.cc M be/src/exec/empty-set-node.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scan-node-mt.h M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node-mt.cc M be/src/exec/kudu-scan-node-mt.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.cc M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/exec/select-node.cc M be/src/exec/select-node.h M be/src/exec/singular-row-src-node.cc M be/src/exec/singular-row-src-node.h M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/streaming-aggregation-node.cc M be/src/exec/streaming-aggregation-node.h M be/src/exec/subplan-node.cc M be/src/exec/subplan-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h 64 files changed, 1,336 insertions(+), 698 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/14764/8 -- To view, visit http://gerrit.cloudera.org:8080/14764 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I69f1676bf67bac31fa5902511b3fcc269fd67472 Gerrit-Change-Number: 14764 Gerrit-PatchSet: 8 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5317/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 21:00:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 9: Code-Review+2 carry from thomas -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 20:59:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 8: (6 comments) http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h File be/src/exec/partitioned-hash-join-builder.h: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h@200 PS8, Line 200: when *repartitioning as true > I think this is supposed to be "when *repartitioned is true"? Done http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h@205 PS8, Line 205: /// 'output_partitions' for build modes like right outer join that output unmatched rows. > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h@213 PS8, Line 213: repartitioning > same - repartitioned? Done http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.cc@641 PS8, Line 641: Pick the next spilled partition the input partition will stay in : // 'spilled_partitions_' until we are done probing it or repartitioning its probe. > Having some trouble parsing this sentence. I think you missed a period afte Done http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-node.cc File be/src/exec/partitioned-hash-join-node.cc: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-node.cc@1091 PS8, Line 1091: // Walk the partitions that had hash tables built for the probe phase and close them. This was also stale http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-node.cc@1098 PS8, Line 1098: // The build partitions we need to retain for further processing. > stale Done -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 20:59:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Hello Thomas Tauber-Marshall, Csaba Ringhofer, Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14790 to look at the new patch set (#9). Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. IMPALA-9126: part 4: hash join builder manages spilling This is the final patch for IMPALA-9126. This will allow the many:1 relationship of probe:build partitions that we need for the shared join build. Key changes: * Builder picks the next spilled partition to process. * Partitions are identified by unique ID so can be decoupled between build and probe. * unique_ptr is used to manage build partitions. This helps document the lifecycle of the partitions better, particularly when they are handed off to PartitionedHashJoinNode. Testing: * Ran exhaustive tests. * Ran a single node TPC-H and TPC-DS stress test with 1000 queries. Perf: Ran a single node TPC-H 30 test against master from before IMPALA-9126 changes. No significant perf change. Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 --- M be/src/exec/partitioned-hash-join-builder-ir.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node-ir.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h 6 files changed, 182 insertions(+), 120 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/90/14790/9 -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7550: Add documentation to profile counters
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14776 ) Change subject: IMPALA-7550: Add documentation to profile counters .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5214/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc03faddb27754001290bb6d899840e2cbe7ccb7 Gerrit-Change-Number: 14776 Gerrit-PatchSet: 6 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 20:46:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7550: Add documentation to profile counters
Jiawei Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14776 ) Change subject: IMPALA-7550: Add documentation to profile counters .. Patch Set 6: (27 comments) Hi Balazs, Thanks so much for help correct the profile counters descriptions! That's really helpful! Agree that we need to keep code out of stable counters descriptions. Unfortunately, I only have so little knowledge on the scan counters so that a lot of counters I am not sure how to fix the description. Ask @Tim Armstrong and @Lars to help take a look on that. Also, if you have strong options how to update the counters. Please feel free to let me know. Have already applied a few suggestions you made. Thanks Jiawei http://gerrit.cloudera.org:8080/#/c/14776/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14776/5//COMMIT_MSG@37 PS5, Line 37: 2. Profile counters are also annotated with their significance to users. : * STABLE_HIGH - High level and stable counters, always useful on measuring : query performance and status. Counters that everyone is interested. should : rarely change and if it does we will make some effort to notify users. : : * STABLE_LOW - Low level and stable counters. Interesting counters to monitor : and analyze by machine. It will probably be interesting under some : circumstance for users. Lots of developers are interested. : : * Unstable - Unstable but useful. Useful to understand query performance, : but subject to change, particularly if the implementation changes. : E.g. MaterializeTupleTimer : : * Debug - Debugging counters. Generally not useful to users of Impala, : the main use case is low-level debugging. Can be hidden to reduce noise : for most consumers of profiles. : : 3. We have around 250 counters. This commit did the replacement in : scan-node and hdfs-scan-node-base and coordinator. : > Please simplify this to what's in the code - I think the descriptions in Si Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc File be/src/exec/hdfs-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@70 PS5, Line 70: across all Disk I/O threads in HDFS read operations."); : PROFILE_DEFINE_TIMER(TotalRawHdfsOpenFileTime, STABLE_LOW, "The tota > Instead of including an explanation in individual description (but not in e Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@79 PS5, Line 79: lectionItemsRead, STABLE_LOW, TUnit::UNIT, > Is this included in profiles where it's irrelevant? If no, remove this to a Not sure about it. @Lars and @Tarmstrong maybe can help to take a look? http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@85 PS5, Line 85: _DEFINE_SA > Maybe '...remote data source...'? Disk queue is not a term users will be fa @Lars and @Tarmstrong http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@88 PS5, Line 88: ty of the sys > Than what? - Is there a point of comparison we can provide? For example, 'v Not sure... if someone have better knowledge than this might want to fix it... http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@89 PS5, Line 89: the s > scan Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@90 PS5, Line 90: it::BYTES, > I don't think we should speculate on root causes - there can be many reason Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@121 PS5, Line 121: the scanner thread was ready to process " : "the data. High values show that scanner threa > Isn't this the same as 'HDFS scans'? IDK to be honest... @Tarmstrong and @Lars http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@125 PS5, Line 125: pressedBytesReadPerColumn, STABLE_LOW, : TUnit::BYTES, "Stats a > replace: Note that this includes the time when Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@130 PS5, Line 130: , "Stats about the number of compressed bytes read per column. " : "Each sample in the counter is the size of a single column that is scanned by the " : "scan node."); > remove? Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@136 PS5, Line 136: of data cache partially hit"); : PROFILE_DEFINE_COUNTER(DataCacheMissCount, STABLE_HIGH, TUnit::UNIT, : "Total count of data > remove? Done http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/scan-node.cc File be/src/exec/scan-node.cc:
[Impala-ASF-CR] IMPALA-7550: Add documentation to profile counters
Hello Andrew Sherman, Lars Volker, David Rorke, Balazs Jeszenszky, Jiawei Wang, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14776 to look at the new patch set (#6). Change subject: IMPALA-7550: Add documentation to profile counters .. IMPALA-7550: Add documentation to profile counters This change changes the way developers define profile counters by generating counters from a counter registry. All the profile counters will be register there first and then used in the same way as before. By doing so, we will be able to manage profile counters in a way that we can define profile counters documentation. For example: Declaration: PROFILE_DEFINE_COUNTER(NumBackends, STABLE_HIGH, TUnit::UNIT, "Number of backends running this query."); Initialization: COUNTER_SET(PROFILE_NumBackends.Instantiate(query_profile_), num_backends); This shall be how we define a NumBackends counter. It follows with its significance, type, description in the declaration part. Users now will be able to view profile counters documentation under query_profile page, there is a Profile Documentation button which leads to /profile_docs. More details: This commit did the following refactors on profile counters. 1. Add a singleton registry for runtime profile counters prototypes, similiar to what Kudu does for metrics. This allows us to generate profile documentation for all counters from the code. We add /profile_docs and a correspoding UI for the documentation of profile counters. 2. Profile counters are also annotated with their significance to users. * STABLE_HIGH - High level and stable counters, always useful on measuring query performance and status. Counters that everyone is interested. should rarely change and if it does we will make some effort to notify users. * STABLE_LOW - Low level and stable counters. Interesting counters to monitor and analyze by machine. It will probably be interesting under some circumstance for users. Lots of developers are interested. * Unstable - Unstable but useful. Useful to understand query performance, but subject to change, particularly if the implementation changes. E.g. MaterializeTupleTimer * Debug - Debugging counters. Generally not useful to users of Impala, the main use case is low-level debugging. Can be hidden to reduce noise for most consumers of profiles. 3. We have around 250 counters. This commit did the replacement in scan-node and hdfs-scan-node-base and coordinator. Concers: The downside is that we will have duplicate comments of query profiles both in the header file and the .cc file. Additionally a (arguably good) limitation is that profile counter names need to be unique. Change-Id: Idc03faddb27754001290bb6d899840e2cbe7ccb7 --- M be/src/exec/hbase-scan-node.cc M be/src/exec/hbase-scan-node.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/kudu-scan-node-base.cc M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator.cc D be/src/util/debug-counters.h M be/src/util/default-path-handlers.cc M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc A www/profile_docs.tmpl M www/queries.tmpl M www/query_profile.tmpl 16 files changed, 605 insertions(+), 273 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/76/14776/6 -- To view, visit http://gerrit.cloudera.org:8080/14776 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idc03faddb27754001290bb6d899840e2cbe7ccb7 Gerrit-Change-Number: 14776 Gerrit-PatchSet: 6 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Balazs Jeszenszky Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Jiawei Wang Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. IMPALA-9217: Adjust limits for TZH and TZM datetime tokens Values for TZH are expected in a range of [-15, 15] and for TZM are expected in [0, 59]. This patch adjusts the checks for these limits and adds some tests to verify that out of range values are not accepted. Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Reviewed-on: http://gerrit.cloudera.org:8080/14848 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/runtime/datetime-iso-sql-format-parser.cc M tests/query_test/test_cast_with_format.py 2 files changed, 19 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 19:58:00 + Gerrit-HasComments: No
[Impala-ASF-CR] Update mvn snapshot dependencies in "clean" checkouts
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14854 ) Change subject: Update mvn snapshot dependencies in "clean" checkouts .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5213/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8bbde17e7c97466391aa20ac3d59c6943e7f7256 Gerrit-Change-Number: 14854 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 19:56:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9126: part 4: hash join builder manages spilling
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14790 ) Change subject: IMPALA-9126: part 4: hash join builder manages spilling .. Patch Set 8: Code-Review+2 (4 comments) http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h File be/src/exec/partitioned-hash-join-builder.h: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h@200 PS8, Line 200: when *repartitioning as true I think this is supposed to be "when *repartitioned is true"? http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.h@213 PS8, Line 213: repartitioning same - repartitioned? http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-builder.cc@641 PS8, Line 641: Pick the next spilled partition the input partition will stay in : // 'spilled_partitions_' until we are done probing it or repartitioning its probe. Having some trouble parsing this sentence. I think you missed a period after "Pick the next spilled partition"? http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-node.cc File be/src/exec/partitioned-hash-join-node.cc: http://gerrit.cloudera.org:8080/#/c/14790/8/be/src/exec/partitioned-hash-join-node.cc@1098 PS8, Line 1098: // The build partitions we need to retain for further processing. stale -- To view, visit http://gerrit.cloudera.org:8080/14790 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6de5f62e3eacf80f72c8ea0ed8cba012f0f53c90 Gerrit-Change-Number: 14790 Gerrit-PatchSet: 8 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 19:35:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Update mvn snapshot dependencies in "clean" checkouts
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14854 Change subject: Update mvn snapshot dependencies in "clean" checkouts .. Update mvn snapshot dependencies in "clean" checkouts buildall.sh saves the cdh/cdp version into .cdh/.cdp, and updates the dependencies if this doesn't match the version from config. This lead to updating the dependencies when switching to a different checkout in the same directory, but didn't do this in a fresh checkout, which could lead to build issues when the .m2 cache was dirty. Note that this doesn't protect from switching between Impala directories with different cdh/cdp versions. Change-Id: I8bbde17e7c97466391aa20ac3d59c6943e7f7256 --- M buildall.sh 1 file changed, 4 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/14854/1 -- To view, visit http://gerrit.cloudera.org:8080/14854 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8bbde17e7c97466391aa20ac3d59c6943e7f7256 Gerrit-Change-Number: 14854 Gerrit-PatchSet: 1 Gerrit-Owner: Csaba Ringhofer
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to ORC scanner
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/14832 ) Change subject: IMPALA-8184: Add timestamp validation to ORC scanner .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 18:16:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14811 ) Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5316/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d Gerrit-Change-Number: 14811 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 17:43:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14852 ) Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5212/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 17:18:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14852 ) Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5211/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 17:15:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. IMPALA-9215: report_benchmark_results.py fails with missing key report_benchmark_results.py failed with missing key because it tried to lookup 'num_instances' from the perf result json file. The JSON file contained exec summary generated by impala_beeswax.py::__build_summary_table() which omitted number of instances. This patch adds 'num_instances' to the summary table created by impala_beeswax.py. To keep report_benchmark_results.py simple it assumes that both perf json files contain 'num_instances', i.e. if a user issues single_node_perf_run.py to compare two commits, both of them must contain this fix. I tested the PS manually. Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Reviewed-on: http://gerrit.cloudera.org:8080/14830 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/beeswax/impala_beeswax.py M tests/benchmark/report_benchmark_results.py 2 files changed, 6 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 17:07:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Attila Jeges has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14852 ) Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 This patch adds week-based date format tokens on top of what was introduced in IMPALA-8703, IMPALA-8704 and IMPALA-8705. Week-based date tokens may be used both for datetime to string and string to datetime conversion. The week-based date tokens are as follows: - IYYY: 4-digit ISO 8601 week-numbering year. Week-numbering year is the year relating to the ISO 8601 week number (IW), which is the full week (Monday to Sunday) which contains January 4 of the Gregorian year. Behaves similarly to in that for datetime to string conversion, prefix digits for 1, 2, and 3-digit inputs are obtained from current week-numbering year. - IYY: Last 3 digits of ISO 8601 week-numbering year. Behaves similarly to YYY in that for datetime to string conversion, prefix digit is obtained from current week-numbering year and can accept 1 or 2-digit input. - IY: Last 2 digits of ISO 8601 week-numbering year. Behaves similarly to YY in that for datetime to string conversion, prefix digits are obtained from current week-numbering year and can accept 1-digit input. - I:Last digit of ISO 8601 week-numbering year. Behaves similarly to Y in that for datetime to string conversion, prefix digits are obtained from current week-numbering year. - IW: ISO 8601 week of year (1-52 or 1-53). Begins on the Monday closest to January 1 of the year. For string to datetime conversion, if the input week does not exist in the input year, an error will be thrown. - ID: ISO 8601 day of week (1-7). 1 means Monday and 7 means Sunday. When doing string to datetime conversion, the week-based tokens are meant to be used together and not mixed with other ISO SQL date tokens. The only exceptions are the day name tokens (DAY and Dy) which may be used instead of ID with the rest of the week-based date tokens. Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b --- M be/src/exprs/date-functions-ir.cc M be/src/runtime/date-parse-util.cc M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M be/src/runtime/datetime-iso-sql-format-parser.cc M be/src/runtime/datetime-iso-sql-format-parser.h M be/src/runtime/datetime-iso-sql-format-tokenizer.cc M be/src/runtime/datetime-parser-common.cc M be/src/runtime/datetime-parser-common.h M be/src/runtime/timestamp-parse-util.cc M be/src/runtime/timestamp-parse-util.h M tests/query_test/test_cast_with_format.py 13 files changed, 847 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/14852/2 -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 2 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9162: Do not apply new inferred predicate to outer joins
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/14813 ) Change subject: IMPALA-9162: Do not apply new inferred predicate to outer joins .. Patch Set 1: Thanks Quanlong for the review and the explanation in the JIRA. Just to be clear you are suggesting to keep my current fix and in addition fix the bug in valueTransferGraph creation ? I could probably re-use the logic that I have for ignoring BinaryPredicates whose SlotRef references the output of an Outer Join. -- To view, visit http://gerrit.cloudera.org:8080/14813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9521bd768c4b333069c34d5c1e11b10ea535827 Gerrit-Change-Number: 14813 Gerrit-PatchSet: 1 Gerrit-Owner: Aman Sinha Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 05 Dec 2019 16:49:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14852 ) Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14852/1/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/14852/1/tests/query_test/test_cast_with_format.py@1438 PS1, Line 1438: d flake8: E303 too many blank lines (2) http://gerrit.cloudera.org:8080/#/c/14852/1/tests/query_test/test_cast_with_format.py@1548 PS1, Line 1548: \ flake8: E502 the backslash is redundant between brackets -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 16:46:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4
Attila Jeges has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14852 Change subject: IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 .. IMPALA-8706: ISO:SQL:2016 datetime patterns - Milestone 4 This patch adds week-based date format tokens on top of what was introduced in IMPALA-8703, IMPALA-8704 and IMPALA-8705. Week-based date tokens may be used both for datetime to string and string to datetime conversion. The week-based date tokens are as follows: - IYYY: 4-digit ISO 8601 week-numbering year. Week-numbering year is the year relating to the ISO 8601 week number (IW), which is the full week (Monday to Sunday) which contains January 4 of the Gregorian year. Behaves similarly to in that for datetime to string conversion, prefix digits for 1, 2, and 3-digit inputs are obtained from current week-numbering year. - IYY: Last 3 digits of ISO 8601 week-numbering year. Behaves similarly to YYY in that for datetime to string conversion, prefix digit is obtained from current week-numbering year and can accept 1 or 2-digit input. - IY: Last 2 digits of ISO 8601 week-numbering year. Behaves similarly to YY in that for datetime to string conversion, prefix digits are obtained from current week-numbering year and can accept 1-digit input. - I:Last digit of ISO 8601 week-numbering year. Behaves similarly to Y in that for datetime to string conversion, prefix digits are obtained from current week-numbering year. - IW: ISO 8601 week of year (1-52 or 1-53). Begins on the Monday closest to January 1 of the year. For string to datetime conversion, if the input week does not exist in the input year, an error will be thrown. - ID: ISO 8601 day of week (1-7). 1 means Monday and 7 means Sunday. When doing string to datetime conversion, the week-based tokens are meant to be used together and not mixed with other ISO SQL date tokens. The only exceptions are the day name tokens (DAY and Dy) which may be used instead of ID with the rest of the week-based date tokens. Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b --- M be/src/exprs/date-functions-ir.cc M be/src/runtime/date-parse-util.cc M be/src/runtime/date-test.cc M be/src/runtime/date-value.cc M be/src/runtime/date-value.h M be/src/runtime/datetime-iso-sql-format-parser.cc M be/src/runtime/datetime-iso-sql-format-parser.h M be/src/runtime/datetime-iso-sql-format-tokenizer.cc M be/src/runtime/datetime-parser-common.cc M be/src/runtime/datetime-parser-common.h M be/src/runtime/timestamp-parse-util.cc M be/src/runtime/timestamp-parse-util.h M tests/query_test/test_cast_with_format.py 13 files changed, 849 insertions(+), 136 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/14852/1 -- To view, visit http://gerrit.cloudera.org:8080/14852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I89a8c1b98742391cb7b331840d216558dbca362b Gerrit-Change-Number: 14852 Gerrit-PatchSet: 1 Gerrit-Owner: Attila Jeges
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5315/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 15:32:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14811 ) Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5210/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d Gerrit-Change-Number: 14811 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 14:42:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7550: Add documentation to profile counters
Balazs Jeszenszky has posted comments on this change. ( http://gerrit.cloudera.org:8080/14776 ) Change subject: IMPALA-7550: Add documentation to profile counters .. Patch Set 5: (28 comments) Looks nice. I haven't looked at the significance fields for individual counters yet. In general, all but DEBUG descriptions should avoid Impala-internal terminology as much as possible. The target audience will never want to open Impala's code, so referencing function names, etc. will be frustrating. Some are harder to word around than others (e.g. scan range), I skipped those for now. http://gerrit.cloudera.org:8080/#/c/14776/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/14776/5//COMMIT_MSG@37 PS5, Line 37: 2. Profile counters are annotated with their stability: : * Stable counters - generally useful to understand query performance, : should only change rarely and if it does we'll make some effort to : notify users. E.g. BytesRead. : * Unstable but useful - useful to understand query performance, but : subject to change, particularly if the implementation changes. E.g. : RowBatchQueuePutWaitTime, MaterializeTupleTimer : * Debugging counters - generally not useful to users of Impala, the main : use case is low-level debugging. Can be hidden to reduce noise for most : consumers of profiles. : : 3. Profile counters are also annotated with their significance to users. : * Critical level counters - always useful on measuring query performance and status. : Counters that everyone are interested. : * High level counters - generally interesting counters. Most of the users will be : interested and all the developers are very interested. : * Medium level counters - somehow interesting counters to monitor. It will probably be : interesting under some circumstance. Lot of developers are interested. : * Low level counters - not interesting to users. Should be useful for developers : to debug only. Please simplify this to what's in the code - I think the descriptions in Significance are a good explanation of the different levels. http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc File be/src/exec/hdfs-scan-node-base.cc: http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@70 PS5, Line 70: by Disk I/O threads in HDFS read operations. For example, if we have 3 reading " : "threads and each spent 1 sec, this counter will report 3 sec.") Instead of including an explanation in individual description (but not in e.g. the next one), I'd stick to a common language that's clear enough by itself for all 'wall clock timers over multiple threads' descriptions. Maybe 'Aggregate wall clock time across all Disk I/O threads...'? http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@79 PS5, Line 79: concept of \"scan range\", including HDFS and Kudu. Is this included in profiles where it's irrelevant? If no, remove this to avoid confusion. If yes, it'd be better to enumerate the full list since 'scan range' is not a user-facing term. http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@85 PS5, Line 85: disk queue Maybe '...remote data source...'? Disk queue is not a term users will be familiar with. Or, again, just enumerate. Is HDFS remote read overall a single disk queue, or individual remote read targets? Same for S3. http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@88 PS5, Line 88: Higher values Than what? - Is there a point of comparison we can provide? For example, 'values close to the number of disks accessed'? http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@89 PS5, Line 89: thread scan http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@90 PS5, Line 90: because of I don't think we should speculate on root causes - there can be many reasons for this. http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@121 PS5, Line 121: traditional HDFS scan nodes and the scan " : "node total time for the MT_DOP > 1 scan nodes Isn't this the same as 'HDFS scans'? http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@122 PS5, Line 122: Low values show Don't scanner threads shut down if there's no work to do? IIUC, slow reads result in low ScannerIoWaitTime and also low AverageNumScannerThreads. If that's correct, I'd just focus on interpreting high values. http://gerrit.cloudera.org:8080/#/c/14776/5/be/src/exec/hdfs-scan-node-base.cc@125 PS5, Line 125: Note that if CPU load is high, this " : "can
[Impala-ASF-CR] IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14714 ) Change subject: IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 .. IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 This patch adds additional datetime format tokens on top of Milestone 1 (IMPALA-8703) and Milestone 2 (IMPALA-8704). The tokens introduced: - Full month name (MONTH, Month, month): In a string to datetime conversion this token can parse textual month name into a datetime type. In a datetime to string conversion this token gives the textual representation of a month. - Short month name (MON, Mon, mon): Similar to the full month name token but this works for 3-character month names like 'JAN'. - Full day name (DAY, Day, day): In a datetime to string conversion this token gives the textual representation of a day like 'Tuesday.' Not suppported in a string to datetime conversion. - Short day name (DY, Dy, dy): Similar to full day name token but this works for 3-character day names like 'TUE'. Not suppported in a string to datetime conversion. - Day of week (D): In a datetime to string conversion this gives a number in [1-7] where 1 represents Sunday. Not supported in a string to datetime conversion. - Quarter of year (Q): In a datetime to string conversion this gives a number in [1-4] representing a quarter of the year. Not supported in a string to datetime conversion. - Week of year (WW): In a datetime to string conversion this gives a number in [1-53] to represent the week of year where the first week starts from 1st of January. Not supported in a string to datetime conversion. - Week of month (W): In a datetime to string conversion this gives a number in [1-5] to represent the week of month where the first week starts from the first day of the month. Not supported in a string to datetime conversion. Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Reviewed-on: http://gerrit.cloudera.org:8080/14714 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/benchmarks/parse-timestamp-benchmark.cc M be/src/common/init.cc M be/src/exprs/date-functions-ir.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc M be/src/exprs/timestamp-functions.h M be/src/runtime/date-parse-util.cc M be/src/runtime/date-parse-util.h M be/src/runtime/datetime-iso-sql-format-parser.cc M be/src/runtime/datetime-iso-sql-format-parser.h M be/src/runtime/datetime-iso-sql-format-tokenizer.cc M be/src/runtime/datetime-parser-common.cc M be/src/runtime/datetime-parser-common.h M be/src/runtime/datetime-simple-date-format-parser.cc M be/src/runtime/datetime-simple-date-format-parser.h M be/src/runtime/timestamp-parse-util.cc M tests/query_test/test_cast_with_format.py 18 files changed, 1,060 insertions(+), 111 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Gerrit-Change-Number: 14714 Gerrit-PatchSet: 12 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14714 ) Change subject: IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 .. Patch Set 11: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Gerrit-Change-Number: 14714 Gerrit-PatchSet: 11 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 14:19:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14811 ) Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file .. Patch Set 8: (2 comments) http://gerrit.cloudera.org:8080/#/c/14811/8/fe/src/main/java/org/apache/impala/analysis/ParquetSchemaExtractor.java File fe/src/main/java/org/apache/impala/analysis/ParquetSchemaExtractor.java: http://gerrit.cloudera.org:8080/#/c/14811/8/fe/src/main/java/org/apache/impala/analysis/ParquetSchemaExtractor.java@113 PS8, Line 113:* https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules-1 line too long (104 > 90) http://gerrit.cloudera.org:8080/#/c/14811/8/fe/src/main/java/org/apache/impala/analysis/ParquetSchemaExtractor.java@186 PS8, Line 186:* https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules line too long (102 > 90) -- To view, visit http://gerrit.cloudera.org:8080/14811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d Gerrit-Change-Number: 14811 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 14:13:12 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Norbert Luksa has posted comments on this change. (
http://gerrit.cloudera.org:8080/14811 )
Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file
..
Patch Set 8:
(16 comments)
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
File fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java:
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java@74
PS7, Line 74: Schema
> nit: Isn't this meant to be Schema
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java@77
PS7, Line 77: Schema
> nit: same as above
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java
File fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java:
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@49
PS7, Line 49:
> nit: Schema
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@49
PS7, Line 49:
> nit: use camel case for acronyms, e.g. OrcSchemaExtractor
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@50
PS7, Line 50:
> I haven't seen any tests for this error msg. Did I miss something?
Tried to create an ORC file with a non-primitive map key, but did not find an
easy way to do so with the available tools.
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@74
PS7, Line 74:
> nit: Orc
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@75
PS7, Line 75:
> Can it really throw AnalysisException?
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@139
PS7, Line 139:
> Might worth a DCHECK here as well that the size of fieldNames equals to the
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@148
PS7, Line 148:
> nit: Orc
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ORCSchemeExtractor.java@170
PS7, Line 170:
> nit: Orc
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ParquetSchemeExtractor.java
File fe/src/main/java/org/apache/impala/analysis/ParquetSchemeExtractor.java:
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/main/java/org/apache/impala/analysis/ParquetSchemeExtractor.java@50
PS7, Line 50:
> nit: Schema
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java:
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2029
PS7, Line 2029:
> To be in line with TestCreateTableLikeFile() could you rename this to TestC
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2029
PS7, Line 2029:
> I'd also add the "file does't exist" coverage to this function.
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2029
PS7, Line 2029: @Test
> nit: put @Test into a separate line
Done
http://gerrit.cloudera.org:8080/#/c/14811/7/fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java@2030
PS7, Line 2030: leLikeFileOrc() throws AnalysisException {
> I though Apache Impala has ORC support regardless of the Hive version. I ca
Right, this comment is misleading. What I meant to write was that the Java API
that comes with the CDH Hive ORC fails getting the schema of the ORC file,
because of a bug in ORC.
This is solved however by listing ORC as an external dependency (like Hive does
from version 3), instead of relying on Hive's ORC version.
(The bug is fixed in later versions.)
http://gerrit.cloudera.org:8080/#/c/14811/7/testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test
File
testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test:
http://gerrit.cloudera.org:8080/#/c/14811/7/testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test@254
PS7, Line 254: --- QUERY
> Why was this parquet test necessary for this ORC related patch?
This test is not related to ORC support, was only at the wrong place (in
create-table-like-file.test)
--
To view, visit http://gerrit.cloudera.org:8080/14811
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
[Impala-ASF-CR] IMPALA-8046: Support CREATE TABLE from an ORC file
Norbert Luksa has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/14811 ) Change subject: IMPALA-8046: Support CREATE TABLE from an ORC file .. IMPALA-8046: Support CREATE TABLE from an ORC file Impala supports creating a table using the schema of a file. However, only Parquet is supported currently. This commit adds support for creating tables from ORC files The change relies on the ORC Java API with version 1.5 or greater, because of a bug in earlier versions. Therefore, ORC is listed as an external dependency, instead of relying on Hive's ORC version (from Hive3, Hive also lists it as a dependency). Also, the commit performs a little clean-up on the ParquetHelper class, renaming it to ParquetSchemaExtractor and removing outdated comments. To create a table from an ORC file, run: CREATE TABLE tablename LIKE ORC '/path/to/file' Tests: * Added analysis tests for primitive and complex types. * Added e2e tests for creating tables from ORC files. Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java A fe/src/main/java/org/apache/impala/analysis/OrcSchemaExtractor.java R fe/src/main/java/org/apache/impala/analysis/ParquetSchemaExtractor.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java A fe/src/main/java/org/apache/impala/util/FileAnalysisUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M impala-parent/pom.xml A testdata/workloads/functional-query/queries/QueryTest/create-table-like-file-orc.test M testdata/workloads/functional-query/queries/QueryTest/create-table-like-file.test M testdata/workloads/functional-query/queries/QueryTest/create-table-like-table.test M tests/common/skip.py M tests/metadata/test_ddl.py 14 files changed, 460 insertions(+), 79 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/14811/8 -- To view, visit http://gerrit.cloudera.org:8080/14811 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I77cd84cda2ed86516937a67eb320fd41e3f1cf2d Gerrit-Change-Number: 14811 Gerrit-PatchSet: 8 Gerrit-Owner: Norbert Luksa Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to ORC scanner
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14832 ) Change subject: IMPALA-8184: Add timestamp validation to ORC scanner .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5209/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 13:40:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 13:18:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to ORC scanner
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/14832 ) Change subject: IMPALA-8184: Add timestamp validation to ORC scanner .. Patch Set 3: Code-Review+1 (6 comments) Carry +1 http://gerrit.cloudera.org:8080/#/c/14832/2/common/thrift/generate_error_codes.py File common/thrift/generate_error_codes.py: http://gerrit.cloudera.org:8080/#/c/14832/2/common/thrift/generate_error_codes.py@443 PS2, Line 443: ORC > nit: ORC Done http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/data/README@456 PS2, Line 456: ORC > nit: ORC Done http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test File testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test: http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test@6 PS2, Line 6: ORC > nit: ORC Done http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test@16 PS2, Line 16: ORC > nit: same here Done http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py@1327 PS2, Line 1327: T > nit: unnecessary ws Done http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py@1328 PS2, Line 1328: tes > nit: you could place it into the previous line Done -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 13:11:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to ORC scanner
Hello Norbert Luksa, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14832 to look at the new patch set (#3). Change subject: IMPALA-8184: Add timestamp validation to ORC scanner .. IMPALA-8184: Add timestamp validation to ORC scanner Hive can write timestamps that are outside Impala's valid range (Impala: 1400- Hive: 0001-). This change adds validation logic to ORC reading that replaces out-of-range timestamps with NULLs and adds a warning to the query. The logic is very similar to the existing validation in Parquet. Some differences: - "time of day" is not checked separately as it doesn't make sense with ORC's encoding - instead of column name only column id is added to the warning Testing: - added a simple EE test that scans an existing ORC file Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 --- M be/src/exec/orc-column-readers.cc M common/thrift/generate_error_codes.py M testdata/data/README A testdata/data/out_of_range_timestamp.orc A testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test M tests/query_test/test_scanners.py 6 files changed, 42 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14832/3 -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 3 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5314/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 12:39:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 12:39:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. Patch Set 1: Code-Review+1 Had a look at other tests in the file, as I see they did not use invalid values. Lgtm. -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 12:33:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14848 ) Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5208/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 11:19:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 11:01:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9217: Adjust limits for TZH and TZM datetime tokens
Gabor Kaszab has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14848 Change subject: IMPALA-9217: Adjust limits for TZH and TZM datetime tokens .. IMPALA-9217: Adjust limits for TZH and TZM datetime tokens Values for TZH are expected in a range of [-15, 15] and for TZM are expected in [0, 59]. This patch adjusts the checks for these limits and adds some tests to verify that out of range values are not accepted. Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 --- M be/src/runtime/datetime-iso-sql-format-parser.cc M tests/query_test/test_cast_with_format.py 2 files changed, 19 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/14848/1 -- To view, visit http://gerrit.cloudera.org:8080/14848 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6fe2534d51396bb5652af6301866e2dd0f3282c2 Gerrit-Change-Number: 14848 Gerrit-PatchSet: 1 Gerrit-Owner: Gabor Kaszab
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to Orc scanner
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/14832 ) Change subject: IMPALA-8184: Add timestamp validation to Orc scanner .. Patch Set 2: Code-Review+1 (4 comments) Found some nits, otherwise lgtm. http://gerrit.cloudera.org:8080/#/c/14832/2/common/thrift/generate_error_codes.py File common/thrift/generate_error_codes.py: http://gerrit.cloudera.org:8080/#/c/14832/2/common/thrift/generate_error_codes.py@443 PS2, Line 443: Orc nit: ORC http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/data/README File testdata/data/README: http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/data/README@456 PS2, Line 456: Orc nit: ORC http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test File testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test: http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test@6 PS2, Line 6: Orc nit: ORC http://gerrit.cloudera.org:8080/#/c/14832/2/testdata/workloads/functional-query/queries/DataErrorsTest/orc-out-of-range-timestamp.test@16 PS2, Line 16: Orc nit: same here -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 10:54:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8184: Add timestamp validation to Orc scanner
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/14832 ) Change subject: IMPALA-8184: Add timestamp validation to Orc scanner .. Patch Set 2: Code-Review+1 (2 comments) http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py@1327 PS2, Line 1327: nit: unnecessary ws http://gerrit.cloudera.org:8080/#/c/14832/2/tests/query_test/test_scanners.py@1328 PS2, Line 1328: """ nit: you could place it into the previous line -- To view, visit http://gerrit.cloudera.org:8080/14832 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ee2ba83a54f93d37e8832e064f2c8418b503490 Gerrit-Change-Number: 14832 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 05 Dec 2019 10:53:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14714 ) Change subject: IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Gerrit-Change-Number: 14714 Gerrit-PatchSet: 11 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 09:55:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14714 ) Change subject: IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 .. Patch Set 11: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5313/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Gerrit-Change-Number: 14714 Gerrit-PatchSet: 11 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 09:55:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/14714 ) Change subject: IMPALA-8705: ISO:SQL:2016 datetime patterns - Milestone 3 .. Patch Set 10: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic797f19a1311b54e5d00d01d0a7afe1f0f21fb8f Gerrit-Change-Number: 14714 Gerrit-PatchSet: 10 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 05 Dec 2019 08:20:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9215: report benchmark results.py fails with missing key
Norbert Luksa has posted comments on this change. ( http://gerrit.cloudera.org:8080/14830 ) Change subject: IMPALA-9215: report_benchmark_results.py fails with missing key .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/14830 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I822c86f621f5a348b56d672c263a2cf9321767ee Gerrit-Change-Number: 14830 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Comment-Date: Thu, 05 Dec 2019 08:13:56 + Gerrit-HasComments: No
