[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15765 ) Change subject: IMPALA-9574: support ubuntu 18.04 base image .. Patch Set 2: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5842/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Apr 2020 05:55:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9646: clean up README
Hello Aman Sinha, David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15719 to look at the new patch set (#3). Change subject: IMPALA-9646: clean up README .. IMPALA-9646: clean up README Misc improvements to get the README up-to-date and direct readers to the most appropriate docs. Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f --- A README-build.md M README.md M bin/rat_exclude_files.txt 3 files changed, 86 insertions(+), 73 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/15719/3 -- To view, visit http://gerrit.cloudera.org:8080/15719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f Gerrit-Change-Number: 15719 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9646: clean up README
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15719 ) Change subject: IMPALA-9646: clean up README .. Patch Set 3: Carry +2. I'm going to manually verify after the automatic checks finish, since only rat-check is really relevant to this change. -- To view, visit http://gerrit.cloudera.org:8080/15719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f Gerrit-Change-Number: 15719 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Apr 2020 05:13:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9646: clean up README
Hello Aman Sinha, David Knupp, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15719 to look at the new patch set (#2). Change subject: IMPALA-9646: clean up README .. IMPALA-9646: clean up README Misc improvements to get the README up-to-date and direct readers to the most appropriate docs. Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f --- A README-build.md M README.md 2 files changed, 85 insertions(+), 72 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/15719/2 -- To view, visit http://gerrit.cloudera.org:8080/15719 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f Gerrit-Change-Number: 15719 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Hello Anurag Mantripragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15765 to look at the new patch set (#2). Change subject: IMPALA-9574: support ubuntu 18.04 base image .. IMPALA-9574: support ubuntu 18.04 base image Automatically detect if we're on Ubuntu 16.04 or 18.04 and use the appropriate base image. Testing: Built an image locally on my Ubuntu 18.04 system and made sure I could start a minicluster and run a query. Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df --- M docker/CMakeLists.txt M docker/impala_base/Dockerfile 2 files changed, 92 insertions(+), 70 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/15765/2 -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15765 ) Change subject: IMPALA-9574: support ubuntu 18.04 base image .. Patch Set 1: You're right, I didn't squash this. -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 21 Apr 2020 05:15:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9646: clean up README
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/15719 )
Change subject: IMPALA-9646: clean up README
..
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15719/1/README-build.md
File README-build.md:
http://gerrit.cloudera.org:8080/#/c/15719/1/README-build.md@38
PS1, Line 38: | PYTHONPATH | Will be changed to include:
"${IMPALA_HOME}/shell/gen-py" "${IMPALA_HOME}/testdata"
"${THRIFT_HOME}/python/lib/python2.7/site-packages" "${HIVE_HOME}/lib/py"
"${IMPALA_HOME}/shell/ext-py/prettytable-0.7.1/dist/prettytable-0.7.1"
"${IMPALA_HOME}/shell/ext-py/sasl-0.1.1/dist/sasl-0.1.1-py2.7-linux-x
"${IMPALA_HOME}/shell/ext-py/sqlparse-0.1.19/dist/sqlparse-0.1.19-py2 |
> Note some of the recent changes from this patch.
Done
--
To view, visit http://gerrit.cloudera.org:8080/15719
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f
Gerrit-Change-Number: 15719
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 21 Apr 2020 05:12:13 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15765 ) Change subject: IMPALA-9574: support ubuntu 18.04 base image .. Patch Set 1: -Verified Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5839/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Apr 2020 03:50:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/15765 ) Change subject: IMPALA-9574: support ubuntu 18.04 base image .. Patch Set 1: Is this a part of a bigger change? I do not see L22 - L48 in asf master branch. -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Apr 2020 03:39:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9574: support ubuntu 18.04 base image
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15765 Change subject: IMPALA-9574: support ubuntu 18.04 base image .. IMPALA-9574: support ubuntu 18.04 base image Automatically detect if we're on Ubuntu 16.04 or 18.04 and use the appropriate base image. Testing: Built an image locally on my Ubuntu 18.04 system and made sure I could start a minicluster and run a query. Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df --- M docker/CMakeLists.txt 1 file changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/15765/1 -- To view, visit http://gerrit.cloudera.org:8080/15765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8dfdb349e78fd76b91138a70449d51b0ef0021df Gerrit-Change-Number: 15765 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 13: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5677/ -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 21 Apr 2020 02:12:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9665: Fixed database not found errors in query test.test insert
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15766 ) Change subject: IMPALA-9665: Fixed database not found errors in query_test.test_insert .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5681/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15766 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd Gerrit-Change-Number: 15766 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Apr 2020 01:52:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9665: Fixed database not found errors in query test.test insert
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/15766 ) Change subject: IMPALA-9665: Fixed database not found errors in query_test.test_insert .. Patch Set 2: Code-Review+2 LGTM -- To view, visit http://gerrit.cloudera.org:8080/15766 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd Gerrit-Change-Number: 15766 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 21 Apr 2020 01:51:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 14: (5 comments) A few minor comments. Looks good overall. http://gerrit.cloudera.org:8080/#/c/13993/14//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13993/14//COMMIT_MSG@11 PS14, Line 11: It can therefore be safely enabled by default. I suppose the enabling by default will be a separate patch ? http://gerrit.cloudera.org:8080/#/c/13993/14//COMMIT_MSG@55 PS14, Line 55: Testing: Thanks for adding the detailed tests :) http://gerrit.cloudera.org:8080/#/c/13993/14/be/src/exec/hdfs-scan-node-mt.cc File be/src/exec/hdfs-scan-node-mt.cc: http://gerrit.cloudera.org:8080/#/c/13993/14/be/src/exec/hdfs-scan-node-mt.cc@98 PS14, Line 98: row_batch->limit_capacity(1); Should this limiting be under an if(is_partition_key_scan_) check ? http://gerrit.cloudera.org:8080/#/c/13993/14/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java File fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java: http://gerrit.cloudera.org:8080/#/c/13993/14/fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java@1134 PS14, Line 1134: long numRangesAdjusted = numRanges * 100; I can see why the adjustment would be needed .. to ensure a distributed scan is done. Presumably, for smaller number of partitions, we would still be ok to run on a single node ? Since this is multiplying a long value, it could overflow in a pathological case, so suggest capping it with max long value. http://gerrit.cloudera.org:8080/#/c/13993/14/testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test File testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test: http://gerrit.cloudera.org:8080/#/c/13993/14/testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test@21 PS14, Line 21:HDFS partitions=24/24 files=24 size=478.45KB it wasn't obvious to me why the HDFS prefix got added based on the changes in this patch since the new item added to the Explain plan is 'partition key scan'. -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Tue, 21 Apr 2020 00:45:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 23:41:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 23:43:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9665: Fixed database not found errors in query test.test insert
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15766 ) Change subject: IMPALA-9665: Fixed database not found errors in query_test.test_insert .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5838/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15766 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd Gerrit-Change-Number: 15766 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 22:59:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9665: Fixed database not found errors in query test.test insert
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/15766 ) Change subject: IMPALA-9665: Fixed database not found errors in query_test.test_insert .. Patch Set 1: Looks good, will wait for build to run. Gerrit says "Cannot Merge" so you may need to rebase. -- To view, visit http://gerrit.cloudera.org:8080/15766 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd Gerrit-Change-Number: 15766 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 22:51:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9646: clean up README
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15719 )
Change subject: IMPALA-9646: clean up README
..
Patch Set 1: Code-Review+2
(1 comment)
Thanks for doing this -- I thin this makes sense.
Note some of the changes to the build page.
http://gerrit.cloudera.org:8080/#/c/15719/1/README-build.md
File README-build.md:
http://gerrit.cloudera.org:8080/#/c/15719/1/README-build.md@38
PS1, Line 38: | PYTHONPATH | Will be changed to include:
"${IMPALA_HOME}/shell/gen-py" "${IMPALA_HOME}/testdata"
"${THRIFT_HOME}/python/lib/python2.7/site-packages" "${HIVE_HOME}/lib/py"
"${IMPALA_HOME}/shell/ext-py/prettytable-0.7.1/dist/prettytable-0.7.1"
"${IMPALA_HOME}/shell/ext-py/sasl-0.1.1/dist/sasl-0.1.1-py2.7-linux-x
"${IMPALA_HOME}/shell/ext-py/sqlparse-0.1.19/dist/sqlparse-0.1.19-py2 |
Note some of the recent changes from this patch.
https://gerrit.cloudera.org/c/15642/10/README.md
--
To view, visit http://gerrit.cloudera.org:8080/15719
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I05fb4a97b6a915fd6e460d9a2079b2d23134678f
Gerrit-Change-Number: 15719
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong
Gerrit-Reviewer: Aman Sinha
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 20 Apr 2020 22:45:24 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9679: Remove some jars from Docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15753 ) Change subject: IMPALA-9679: Remove some jars from Docker images .. Patch Set 5: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5837/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 Gerrit-Change-Number: 15753 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 22:32:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9665: Fixed database not found errors in query test.test insert
Adam Tamas has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15766 Change subject: IMPALA-9665: Fixed database not found errors in query_test.test_insert .. IMPALA-9665: Fixed database not found errors in query_test.test_insert Fixed the usage of the unique_database in the test_insert.py to wait with the tests until the database is synced. Testing: -tests/run-tests.py query_test/test_insert.py --exploration_strategy=exhaustive Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd --- M tests/query_test/test_insert.py 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/15766/1 -- To view, visit http://gerrit.cloudera.org:8080/15766 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9b7aa3775dd4375f536d76f2e236ce126f8c78cd Gerrit-Change-Number: 15766 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas
[Impala-ASF-CR] IMPALA-9679: Remove some jars from Docker images
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15753 ) Change subject: IMPALA-9679: Remove some jars from Docker images .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15753/5/docker/setup_build_context.py File docker/setup_build_context.py: http://gerrit.cloudera.org:8080/#/c/15753/5/docker/setup_build_context.py@95 PS5, Line 95: ) flake8: E501 line too long (92 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/15753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 Gerrit-Change-Number: 15753 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 21:52:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9679: Remove some jars from Docker images
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15753 to look at the new patch set (#5). Change subject: IMPALA-9679: Remove some jars from Docker images .. IMPALA-9679: Remove some jars from Docker images This removes a few transitive dependencies that don't appear to be needed at runtime. This also removes the frontend test jar. The inclusion of that jar was masking an issue where some configs were not accessible from within the container, because they were symlinks to paths on the host. Testing: Ran dockerized tests in precommit. Ran regular tests with CDP hive. Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 --- M bin/create-test-configuration.sh M docker/setup_build_context.py M fe/pom.xml 3 files changed, 60 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/15753/5 -- To view, visit http://gerrit.cloudera.org:8080/15753 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I030e7cd28e29cd4e077c0b4addd4d14a8599eed6 Gerrit-Change-Number: 15753 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15760/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15760/1//COMMIT_MSG@9 PS1, Line 9: fluent-hc has been flagged in CVE-2014-3577 and CVE-2015-5262, and : doesn't seem to be a required component for us. > The commit message needs to include information about how the code change w Thanks for the tip about adding Impala Public Jenkins as a reviewer. I've never been able to figure out how to test draft patches with jenkins.impala.io, and had assumed it was not possible. I'll amend the commit message when then test run completes. -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 21:26:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 14: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5836/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 14 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 20 Apr 2020 21:11:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9668: Obey SKIP TOOLCHAIN BOOTSTRAP during virtualenv bootstrap
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15759 ) Change subject: IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 Gerrit-Change-Number: 15759 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 20:40:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode.
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15752 ) Change subject: IMPALA-9540 Test that Impala Shell no longer sends duplicate "Host" headers in http mode. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15752/1/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/15752/1/tests/shell/test_shell_interactive.py@88 PS1, Line 88: self.headers.headers Sorry for this potentially obstuse question, but I have to confess, I'm a little perplexed by this. "headers" is an attribute that itself has an attribute called "headers"? It looks like headers is just a run-of-the-mill list. Is the "self" object in this case an instance of the parent class. -- To view, visit http://gerrit.cloudera.org:8080/15752 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82996015d0205923e854dac8bb88604778684c46 Gerrit-Change-Number: 15752 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 20:28:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Hello Quanlong Huang, Aman Sinha, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13993 to look at the new patch set (#14). Change subject: IMPALA-8834: Short-circuit partition key scan .. IMPALA-8834: Short-circuit partition key scan This generalises the OPTIMIZE_PARTITION_KEY_SCANS to return correct results when files have zero rows. It can therefore be safely enabled by default. Instead of evaluating the query using metadata, the planner instructs the backend to short-circuit HDFS scans after a single row has been returned from each file. This gives results equivalent to returning all the rows from each file. Planner cardinality estimates are adjusted accordingly to enable potentially better plans and other optimisations like disabling codegen. We make some effort to avoid generated extra scan ranges for remote scans by only generating one range per remote file. The backend optimisation is implemented by constructing a row batch with capacity for a single row only and then terminating each scan range once a single row has been produced. Both Parquet and ORC have optimized code paths for zero slot table scans that mean this will only result in a footer read. (Other file formats still need to read some portion of the file, but can terminate early once one row has been produced.) This should be quite efficient in practice with file handle caching and data caching enabled, because it then only requires reading the footer from the cache for each file. The partition key scan optimization is also slightly generalised to apply to scans of unpartitioned tables where no slots are materialized. A limitation of the optimization where it did not apply to multiple grouping classes was also fixed. Limitations: * This still scans every file in the partition. I.e. there is no short-circuiting if a row has already been found in the partition by the current scan node. * Resource reservations and estimates for the scan node do not all take into account this optimisation, so are conservative - they assume the whole file is scanned. Testing: * Added end-to-end tests that execute the query on all HDFS file formats and verify that the correct number of rows flow through the plan. * Added planner test based on the existing test partition key scan test. * Added test to make sure single node optimisation kicks in when expected. * Add test for cardinality estimates with and without stats * Added test for unpartitioned tables. * Added planner test that checks that optimisation is enabled for multiple aggregation classes. * Added a targeted perf test. Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/distinct.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test A testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans-default.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/shuffle-by-distinct-exprs.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans-with-joins.test A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans.test M testdata/workloads/targeted-perf/queries/aggregation.test M tests/query_test/test_queries.py 32 files changed, 1,162 insertions(+), 231 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13993/14 -- To view, visit
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 13: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5834/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 20 Apr 2020 20:03:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5835/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 20:04:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15760/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15760/1//COMMIT_MSG@9 PS1, Line 9: fluent-hc has been flagged in CVE-2014-3577 and CVE-2015-5262, and : doesn't seem to be a required component for us. The commit message needs to include information about how the code change was tested. The expectation is that the code change was tested prior to code review. If you want to test using jenkins.impala.io without it being visible to reviewers, one option is to use gerrit drafts (including Impala Public Jenkins as a reviewer so it has access). -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 19:34:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5833/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 19:17:20 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 13: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5677/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 20 Apr 2020 19:22:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15761 ) Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5679/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 19:26:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15760 ) Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5678/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 20 Apr 2020 19:26:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9648: Exclude netty from hadoop-hdfs mvn download
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15761 Change subject: IMPALA-9648: Exclude netty from hadoop-hdfs mvn download .. IMPALA-9648: Exclude netty from hadoop-hdfs mvn download Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 --- M fe/pom.xml 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/15761/1 -- To view, visit http://gerrit.cloudera.org:8080/15761 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I69bc84eaa692e78988cff06bb1b7e5fd2a0bb774 Gerrit-Change-Number: 15761 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Patch Set 13: (6 comments) http://gerrit.cloudera.org:8080/#/c/13993/13/fe/src/main/java/org/apache/impala/planner/ScanNode.java File fe/src/main/java/org/apache/impala/planner/ScanNode.java: http://gerrit.cloudera.org:8080/#/c/13993/13/fe/src/main/java/org/apache/impala/planner/ScanNode.java@69 PS13, Line 69: // The caller passes in a MultiAggregateInfo to the constructor that this scan node uses to line too long (93 > 90) http://gerrit.cloudera.org:8080/#/c/13993/13/fe/src/main/java/org/apache/impala/planner/ScanNode.java@71 PS13, Line 71: // applied to the MultiAggregateInfo in this query block. We do not apply the smap in this line too long (92 > 90) http://gerrit.cloudera.org:8080/#/c/13993/13/tests/query_test/test_queries.py File tests/query_test/test_queries.py: http://gerrit.cloudera.org:8080/#/c/13993/13/tests/query_test/test_queries.py@246 PS13, Line 246: class TestPartitionKeyScans(ImpalaTestSuite): flake8: E302 expected 2 blank lines, found 1 http://gerrit.cloudera.org:8080/#/c/13993/13/tests/query_test/test_queries.py@254 PS13, Line 254: \ flake8: E502 the backslash is redundant between brackets http://gerrit.cloudera.org:8080/#/c/13993/13/tests/query_test/test_queries.py@257 PS13, Line 257: flake8: E203 whitespace before ':' http://gerrit.cloudera.org:8080/#/c/13993/13/tests/query_test/test_queries.py@257 PS13, Line 257: : flake8: E231 missing whitespace after ':' -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 20 Apr 2020 19:22:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Tim Armstrong has removed Michael Ho from this change. ( http://gerrit.cloudera.org:8080/13993 ) Change subject: IMPALA-8834: Short-circuit partition key scan .. Removed reviewer Michael Ho. -- To view, visit http://gerrit.cloudera.org:8080/13993 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c Gerrit-Change-Number: 13993 Gerrit-PatchSet: 13 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan
Hello Michael Ho, Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13993 to look at the new patch set (#13). Change subject: IMPALA-8834: Short-circuit partition key scan .. IMPALA-8834: Short-circuit partition key scan This generalises the OPTIMIZE_PARTITION_KEY_SCANS to return correct results when files have zero rows. It can therefore be safely enabled by default. Instead of evaluating the query using metadata, the planner instructs the backend to short-circuit HDFS scans after a single row has been returned from each file. This gives results equivalent to returning all the rows from each file. Planner cardinality estimates are adjusted accordingly to enable potentially better plans and other optimisations like disabling codegen. We make some effort to avoid generated extra scan ranges for remote scans by only generating one range per remote file. The backend optimisation is implemented by constructing a row batch with capacity for a single row only and then terminating each scan range once a single row has been produced. Both Parquet and ORC have optimized code paths for zero slot table scans that mean this will only result in a footer read. (Other file formats still need to read some portion of the file, but can terminate early once one row has been produced.) This should be quite efficient in practice with file handle caching and data caching enabled, because it then only requires reading the footer from the cache for each file. The partition key scan optimization is also slightly generalised to apply to scans of unpartitioned tables where no slots are materialized. A limitation of the optimization where it did not apply to multiple grouping classes was also fixed. Limitations: * This still scans every file in the partition. I.e. there is no short-circuiting if a row has already been found in the partition by the current scan node. * Resource reservations and estimates for the scan node do not all take into account this optimisation, so are conservative - they assume the whole file is scanned. Testing: * Added end-to-end tests that execute the query on all HDFS file formats and verify that the correct number of rows flow through the plan. * Added planner test based on the existing test partition key scan test. * Added test to make sure single node optimisation kicks in when expected. * Add test for cardinality estimates with and without stats * Added test for unpartitioned tables. * Added planner test that checks that optimisation is enabled for multiple aggregation classes. * Added a targeted perf test. Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-scan-node-mt.cc M be/src/exec/hdfs-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/runtime/row-batch.cc M be/src/runtime/row-batch.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java M fe/src/main/java/org/apache/impala/planner/ScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java M fe/src/test/java/org/apache/impala/planner/ExplainTest.java M fe/src/test/java/org/apache/impala/planner/PlannerTest.java M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test M testdata/workloads/functional-planner/queries/PlannerTest/distinct.test M testdata/workloads/functional-planner/queries/PlannerTest/kudu-stats-agg.test M testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-materialization.test M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test A testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans-default.test M testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans.test M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test M testdata/workloads/functional-planner/queries/PlannerTest/shuffle-by-distinct-exprs.test M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans-with-joins.test A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans.test M testdata/workloads/targeted-perf/queries/aggregation.test M tests/query_test/test_queries.py 32 files changed, 1,159 insertions(+), 229 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13993/13 -- To view, visit
[Impala-ASF-CR] WIP IMPALA-9434: Implement Robin Hood Hash Table.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15511 ) Change subject: WIP IMPALA-9434: Implement Robin Hood Hash Table. .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5832/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28eeccd7f9ccae39e31972391f971901bcbfe986 Gerrit-Change-Number: 15511 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 19:13:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15683 ) Change subject: IMPALA-3741 [part 2]: Push runtime bloom filter to Kudu .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/15683/4/common/thrift/PlanNodes.thrift File common/thrift/PlanNodes.thrift: http://gerrit.cloudera.org:8080/#/c/15683/4/common/thrift/PlanNodes.thrift@124 PS4, Line 124: MIN_MAX = 2 > The code for showing the current settings in impala-shell are auto generate It's possible that the comma separated list thing is a bad idea and won't work, but I'm not sure why the shell would be affected by it. I think it should be possible to just treat is as a string from the perspective of impala-shell, its only necessary to break it up into a list internally in Impala. -- To view, visit http://gerrit.cloudera.org:8080/15683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9100076f68ea299ddb6ec8bc027cac7a47f5d754 Gerrit-Change-Number: 15683 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Bankim Bhavsar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Mon, 20 Apr 2020 18:56:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-9434: Implement Robin Hood Hash Table.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/15511 ) Change subject: WIP IMPALA-9434: Implement Robin Hood Hash Table. .. Patch Set 12: New change in Patch Set 12: - Instead of measuring fill ratio as percentage, count the number of filled bucket instead (counter FilledBuckets). - Add InsertTravel counter to measure how much travel involved in insertion path. Consequently, number of travel in lookup-only can be computed as (Travel - InsertTravel). - Try to speedup bucket rebalancing process by displacing buckets altogether instead of swapping one-by-one. -- To view, visit http://gerrit.cloudera.org:8080/15511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28eeccd7f9ccae39e31972391f971901bcbfe986 Gerrit-Change-Number: 15511 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 18:36:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml
David Knupp has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15760 Change subject: IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml .. IMPALA-9647: Exclude fluent-hc-4.3.2.jar from fe/pom.xml fluent-hc has been flagged in CVE-2014-3577 and CVE-2015-5262, and doesn't seem to be a required component for us. Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 --- M fe/pom.xml 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/15760/1 -- To view, visit http://gerrit.cloudera.org:8080/15760 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I245ddb50e17d49295b1a1ebaef6c17e1b92304f1 Gerrit-Change-Number: 15760 Gerrit-PatchSet: 1 Gerrit-Owner: David Knupp
[Impala-ASF-CR] WIP IMPALA-9434: Implement Robin Hood Hash Table.
Hello David Rorke, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15511 to look at the new patch set (#12). Change subject: WIP IMPALA-9434: Implement Robin Hood Hash Table. .. WIP IMPALA-9434: Implement Robin Hood Hash Table. Robin hood hashing reduces the variances of probe lengths by continually rebalancing elements. This patch is the first step towards full robin hood hash table implementation by doing bucket rebalancing after every insert. If a hash table is configured as a robin hood hash table, the new element insertion will be buffered in a temporary bucket. This temporary bucket will then matched against existing bucket elements, swapped with a "rich" bucket, and continue doing so until it swap with an empty bucket. The PSL (probe sequence length) invariant is maintained in robin hood hash table. This allow us to add short-circuit in the Probe function to immediately returns when it finds out that the PSL of currently visited bucket is smaller/richer than the key that is being looked up, indicating that the key does not exist in the table. Instead of continue probing until next empty bucket is found, Probe can immediately the return the index of recently visited richer bucket to caller along with the not found flag. In turn, the caller use this returned index to specify in which index the new element should be inserted to. Testing: - Add backend test for Robin Hood hash table in hash-table-test.cc - Pass core tests Change-Id: I28eeccd7f9ccae39e31972391f971901bcbfe986 --- M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/hash-table-benchmark.cc M be/src/exec/grouping-aggregator-ir.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hash-table-test.cc M be/src/exec/hash-table.cc M be/src/exec/hash-table.h M be/src/exec/hash-table.inline.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-node.cc M be/src/exprs/scalar-expr.h 12 files changed, 832 insertions(+), 92 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/11/15511/12 -- To view, visit http://gerrit.cloudera.org:8080/15511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I28eeccd7f9ccae39e31972391f971901bcbfe986 Gerrit-Change-Number: 15511 Gerrit-PatchSet: 12 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9666 Correct a log message in TestImpalaShellInteractive
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15751 ) Change subject: IMPALA-9666 Correct a log message in TestImpalaShellInteractive .. IMPALA-9666 Correct a log message in TestImpalaShellInteractive When _wait_for_num_open_sessions() calls wait_for_metric_value() and wait_for_metric_value() hits an assertion error then _wait_for_num_open_sessions() catches the exception and attempts to log a descriptive string that was passed to it as a parameter. The logging call does not have a '%s' directive, which results in the failure being reported as "TypeError: not all arguments converted during string formatting". Fix this by adding the '%s' directive to the logging call. Change-Id: Icfd7baed153dadceb953df30beba180128055c92 Reviewed-on: http://gerrit.cloudera.org:8080/15751 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- 0 files changed, 0 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icfd7baed153dadceb953df30beba180128055c92 Gerrit-Change-Number: 15751 Gerrit-PatchSet: 3 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9666 Correct a log message in TestImpalaShellInteractive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15751 ) Change subject: IMPALA-9666 Correct a log message in TestImpalaShellInteractive .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfd7baed153dadceb953df30beba180128055c92 Gerrit-Change-Number: 15751 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 16:38:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9666 Correct a log message in TestImpalaShellInteractive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15751 ) Change subject: IMPALA-9666 Correct a log message in TestImpalaShellInteractive .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfd7baed153dadceb953df30beba180128055c92 Gerrit-Change-Number: 15751 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 16:30:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9666 Correct a log message in TestImpalaShellInteractive
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15751 ) Change subject: IMPALA-9666 Correct a log message in TestImpalaShellInteractive .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/365/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15751 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfd7baed153dadceb953df30beba180128055c92 Gerrit-Change-Number: 15751 Gerrit-PatchSet: 2 Gerrit-Owner: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 16:30:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. Patch Set 3: (5 comments) http://gerrit.cloudera.org:8080/#/c/15746/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15746/2//COMMIT_MSG@7 PS2, Line 7: IMPALA-9631: Import HLL functionality from DataSketches > Do you plan to merge this patch as it is? I would consider it more of a POC Well, I don't see a difference of merging it now of waiting for other development work and start the review then. Btw, this unlock other steps under the umbrella Jira: https://issues.apache.org/jira/browse/IMPALA-9593 Additionally, waiting for the rest of the functionality means that splitting up these tasks has no point and we could have kept a single Jira for everything HLL related. So I'd take this as a non-POC review now and we still can make changes later on if we want to modify the build structure or drop the entire library for some reason. http://gerrit.cloudera.org:8080/#/c/15746/2//COMMIT_MSG@9 PS2, Line 9: This patch imports the functionality needed for HLL approximate > Can you add this information in a README.md? To the datasketches folder? Done http://gerrit.cloudera.org:8080/#/c/15746/2//COMMIT_MSG@11 PS2, Line 11: files into be/src/thirdparty/datasketches. Note, that the original > nit: wrap at 72 Done http://gerrit.cloudera.org:8080/#/c/15746/2/be/src/exprs/datasketches-test.cc File be/src/exprs/datasketches-test.cc: http://gerrit.cloudera.org:8080/#/c/15746/2/be/src/exprs/datasketches-test.cc@27 PS2, Line 27: algorith > typo: algorithm Done http://gerrit.cloudera.org:8080/#/c/15746/2/be/src/exprs/datasketches-test.cc@38 PS2, Line 38: std::stringstream sketch_stream1; : std::stringstream sketch_stream2; > I would prefer to use stringstream instead of files to make this faster/avo Done -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 15:26:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9668: Obey SKIP TOOLCHAIN BOOTSTRAP during virtualenv bootstrap
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15759 ) Change subject: IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5831/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 Gerrit-Change-Number: 15759 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Anonymous Coward Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 20 Apr 2020 16:05:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15746 ) Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. Patch Set 3: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/5830/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 20 Apr 2020 15:54:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9668: Obey SKIP TOOLCHAIN BOOTSTRAP during virtualenv bootstrap
Laszlo Gaal has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15759 Change subject: IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap .. IMPALA-9668: Obey SKIP_TOOLCHAIN_BOOTSTRAP during virtualenv bootstrap IMPALA-9626 broke the use case where the toolchain binaries are not downloaded from the native-toolchain S3 bucket, because SKIP_TOOLCHAIN_BOOTSTRAP is set to true. Fix this use case by checking SKIP_TOOLCHAIN_BOOTSTRAP in bin/bootstrap_environment.py: - if true: just check if the specified version of the Python binary is present at the expected toolchain location. If it is there, use it, otherwise throw an exception and abort the bootstrap process. - in any other case: proceed to download the Python binary as in bootstrap_toolchain.py. Test: - simulate the custom toolchain setup by downloading the toolchain binaries from the S3 bucket, copying them to a separate directory, symlinking them into Impala/toolchain, then executing buildall.sh with SKIP_BOOTSTRAP_TOOLCHAIN set to "true". Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 --- M bin/bootstrap_toolchain.py M infra/python/bootstrap_virtualenv.py 2 files changed, 8 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/15759/1 -- To view, visit http://gerrit.cloudera.org:8080/15759 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic51b3c327b3cebc08edff90de931d07e35e0c319 Gerrit-Change-Number: 15759 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/15746 )
Change subject: IMPALA-9631: Import HLL functionality from DataSketches
..
Patch Set 3:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h
File be/src/thirdparty/datasketches/MurmurHash3.h:
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@29
PS3, Line 29: #define FORCE_INLINE __forceinline
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@33
PS3, Line 33: #define ROTL32(x,y) _rotl(x,y)
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@34
PS3, Line 34: #define ROTL64(x,y) _rotl64(x,y)
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@44
PS3, Line 44: #define FORCE_INLINE inline __attribute__((always_inline))
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@56
PS3, Line 56: #define ROTL32(x,y) rotl32(x,y)
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@57
PS3, Line 57: #define ROTL64(x,y) rotl64(x,y)
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@97
PS3, Line 97: FORCE_INLINE void MurmurHash3_x64_128(const void* key, int
lenBytes, uint64_t seed, HashState& out) {
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/15746/3/be/src/thirdparty/datasketches/MurmurHash3.h@108
PS3, Line 108: const int nblocks = lenBytes >> 4; // bytes / 16
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/15746
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf
Gerrit-Change-Number: 15746
Gerrit-PatchSet: 3
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Mon, 20 Apr 2020 15:14:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9631: Import HLL functionality from DataSketches
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15746 to look at the new patch set (#3). Change subject: IMPALA-9631: Import HLL functionality from DataSketches .. IMPALA-9631: Import HLL functionality from DataSketches This patch imports the functionality needed for HLL approximate algorithm from Apache DataSketches. I decided to copy the necessary files into be/src/thirdparty/datasketches. Note, that the original structure of files was changed during this process as originally hll/ and common/ libraries were both affected but I copied these into the same directory so that Impala can compile them without rewriting the include paths in the files themselves. Also note, that not the whole common/ directory was copied just the files needed for HLL. The git hash of the snapshot I used as a source for the files: a6265b307a03085abe26c20413fdbf7d7a5eaf29 Browse the source files here: https://github.com/apache/incubator-datasketches-cpp Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf --- M .clang-tidy M be/src/exprs/CMakeLists.txt A be/src/exprs/datasketches-test.cc A be/src/thirdparty/datasketches/AuxHashMap-internal.hpp A be/src/thirdparty/datasketches/AuxHashMap.hpp A be/src/thirdparty/datasketches/CommonUtil.hpp A be/src/thirdparty/datasketches/CompositeInterpolationXTable-internal.hpp A be/src/thirdparty/datasketches/CompositeInterpolationXTable.hpp A be/src/thirdparty/datasketches/CouponHashSet-internal.hpp A be/src/thirdparty/datasketches/CouponHashSet.hpp A be/src/thirdparty/datasketches/CouponList-internal.hpp A be/src/thirdparty/datasketches/CouponList.hpp A be/src/thirdparty/datasketches/CubicInterpolation-internal.hpp A be/src/thirdparty/datasketches/CubicInterpolation.hpp A be/src/thirdparty/datasketches/HarmonicNumbers-internal.hpp A be/src/thirdparty/datasketches/HarmonicNumbers.hpp A be/src/thirdparty/datasketches/Hll4Array-internal.hpp A be/src/thirdparty/datasketches/Hll4Array.hpp A be/src/thirdparty/datasketches/Hll6Array-internal.hpp A be/src/thirdparty/datasketches/Hll6Array.hpp A be/src/thirdparty/datasketches/Hll8Array-internal.hpp A be/src/thirdparty/datasketches/Hll8Array.hpp A be/src/thirdparty/datasketches/HllArray-internal.hpp A be/src/thirdparty/datasketches/HllArray.hpp A be/src/thirdparty/datasketches/HllSketch-internal.hpp A be/src/thirdparty/datasketches/HllSketchImpl-internal.hpp A be/src/thirdparty/datasketches/HllSketchImpl.hpp A be/src/thirdparty/datasketches/HllSketchImplFactory.hpp A be/src/thirdparty/datasketches/HllUnion-internal.hpp A be/src/thirdparty/datasketches/HllUtil.hpp A be/src/thirdparty/datasketches/LICENSE A be/src/thirdparty/datasketches/MurmurHash3.h A be/src/thirdparty/datasketches/README.md A be/src/thirdparty/datasketches/RelativeErrorTables-internal.hpp A be/src/thirdparty/datasketches/RelativeErrorTables.hpp A be/src/thirdparty/datasketches/coupon_iterator-internal.hpp A be/src/thirdparty/datasketches/coupon_iterator.hpp A be/src/thirdparty/datasketches/hll.hpp A be/src/thirdparty/datasketches/hll.private.hpp A be/src/thirdparty/datasketches/inv_pow2_table.hpp M bin/rat_exclude_files.txt 41 files changed, 7,379 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/15746/3 -- To view, visit http://gerrit.cloudera.org:8080/15746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8ca8e77dcbb6b6c3b1e3bca7ab57cb7d3c018bbf Gerrit-Change-Number: 15746 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15737 ) Change subject: IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5829/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15737 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6a3a502fb3b3cfe7a68323d3b0145e5fb149460 Gerrit-Change-Number: 15737 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Apr 2020 14:40:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs
Hello Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15737 to look at the new patch set (#3). Change subject: IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs .. IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs This commit eliminates an unnecessary hdfsDelete() call for INSERT-only tables. For such tables we don't create a staging directory, hence there is nothing to delete. It also eliminates a confusing log message about the deletion of the staging directory (which did not even exist). I also added a DCHECK to check that we don't even set the staging directory for such tables. Change-Id: Ie6a3a502fb3b3cfe7a68323d3b0145e5fb149460 --- M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M fe/src/main/java/org/apache/impala/service/Frontend.java 3 files changed, 19 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15737/3 -- To view, visit http://gerrit.cloudera.org:8080/15737 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie6a3a502fb3b3cfe7a68323d3b0145e5fb149460 Gerrit-Change-Number: 15737 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15737 ) Change subject: IMPALA-9653: Impala shouldn't create/remove staging directory during transactional INSERTs .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/15737/1/fe/src/main/java/org/apache/impala/service/Frontend.java File fe/src/main/java/org/apache/impala/service/Frontend.java: http://gerrit.cloudera.org:8080/#/c/15737/1/fe/src/main/java/org/apache/impala/service/Frontend.java@1703 PS1, Line 1703: // TODO: Currently this flag only controls the removal of the query-level staging : // directory. HdfsTableSink (that creates the staging dir) c > My idea here is not to fix it, but to remove this member from thrift, as it Added TODO comment. -- To view, visit http://gerrit.cloudera.org:8080/15737 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie6a3a502fb3b3cfe7a68323d3b0145e5fb149460 Gerrit-Change-Number: 15737 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Apr 2020 13:59:15 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9469: ORC scanner vectorization for collection types
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15730 ) Change subject: IMPALA-9469: ORC scanner vectorization for collection types .. IMPALA-9469: ORC scanner vectorization for collection types This commit only keeps the batched read path of ORC columns, i.e. from now on we always read ORC values into a scratch batch. Thanks to this we also get codegen out of the box. >From now on materialization of the table-level tuples are always driven by the root struct reader. This will enable us to implement row validation (against a valid write id list) much easier. It's needed for IMPALA-9512. I eliminated the OrcComplexColumnReader::TransferTuple() interface and the related codes. HdfsOrcScanner became simpler. Now it just calls TopLevelReadValueBatch() on the root struct reader which tracks the row index of the table-level tuples and calls ReadValueBatch on its children accordingly. The children don't need to track the state as they are always being told which row they need to read. Testing: * ran exhaustive tests Performance: * non-nested benchmark results stayed the same as expected * Overall 1-2% gain on TPCH Nested, scale=1 ** In some cases scanning was ~20% more efficient Change-Id: I477961b427406035a04529c5175dbee8f8a93ad5 Reviewed-on: http://gerrit.cloudera.org:8080/15730 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h 4 files changed, 225 insertions(+), 263 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15730 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I477961b427406035a04529c5175dbee8f8a93ad5 Gerrit-Change-Number: 15730 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9469: ORC scanner vectorization for collection types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15730 ) Change subject: IMPALA-9469: ORC scanner vectorization for collection types .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15730 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I477961b427406035a04529c5175dbee8f8a93ad5 Gerrit-Change-Number: 15730 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Apr 2020 13:12:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9469: ORC scanner vectorization for collection types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15730 ) Change subject: IMPALA-9469: ORC scanner vectorization for collection types .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5674/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15730 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I477961b427406035a04529c5175dbee8f8a93ad5 Gerrit-Change-Number: 15730 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Apr 2020 08:55:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9469: ORC scanner vectorization for collection types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15730 ) Change subject: IMPALA-9469: ORC scanner vectorization for collection types .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15730 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I477961b427406035a04529c5175dbee8f8a93ad5 Gerrit-Change-Number: 15730 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Norbert Luksa Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 20 Apr 2020 08:55:55 + Gerrit-HasComments: No
