[Impala-ASF-CR] IMPALA-9398: Fix shell history duplication when cmdloop breaks
Tamas Mate has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15345 Change subject: IMPALA-9398: Fix shell history duplication when cmdloop breaks .. IMPALA-9398: Fix shell history duplication when cmdloop breaks This change adds a new condition to avoid re-reading the impala-shell history when the cmdloop is broken. The loop can break due to exceptions such as KeyboardInterrupt. Testing: - The change was testad manually on my local dev env - Added a new EE shell test to verify the history after SIGINT Change-Id: If4faf46134f44d91e56748642f47d448707db53c --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 36 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/15345/1 -- To view, visit http://gerrit.cloudera.org:8080/15345 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: If4faf46134f44d91e56748642f47d448707db53c Gerrit-Change-Number: 15345 Gerrit-PatchSet: 1 Gerrit-Owner: Tamas Mate
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5398/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 3 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 06:14:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. IMPALA-9452: slightly reduce reservation for test_spilling_aggs As far as I can tell, the query failed to spill because the pre-agg was able to release reservation before the post-agg needed it. Probably there is some variance because of buffering in the exchange. This change slightly reduces the reservation to minimise the chance of this recurring. Also remove a duplicated instance of this test. Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Reviewed-on: http://gerrit.cloudera.org:8080/15339 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test 1 file changed, 1 insertion(+), 24 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 06:12:23 + Gerrit-HasComments: No
[Impala-ASF-CR] santizer and -release flags should cause buildall.sh to exit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15341 ) Change subject: santizer and -release flags should cause buildall.sh to exit .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e Gerrit-Change-Number: 15341 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 06:02:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5397/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 2 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 05:44:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 05:37:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created Currently the exprs created in plan nodes and sink configs were being closed in their respective exec nodes and data sink instances (that were created using them) when the fragment instance closes. This patch ensure that they are closed from the plan nodes and sink configs themselves so that when these are shared among instances, they don't prematurely close the expressions. Testing: Successfully passed exhaustive tests. Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Reviewed-on: http://gerrit.cloudera.org:8080/15318 Reviewed-by: Bikramjeet Vig Tested-by: Impala Public Jenkins --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/scan-node.cc M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M be/src/runtime/data-stream-test.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h 41 files changed, 225 insertions(+), 139 deletions(-) Approvals: Bikramjeet Vig: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 5 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Hello Andrew Sherman, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15219 to look at the new patch set (#3). Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode In order to improve usability, this patch would like to make Impala shell show query processing status while the query is running. The patch enables shell option live_progress by default when a user launches impala shell in the interactive mode. The patch also provides a new command line flag "--disable_live_progress", which allows a user to disable live_progress at runtime. In the interactive mode, a user can disable live_progress by either using the proposed command line flag or setting the option as False in the config file. As for in the non-interactive mode (when the -q or -f options are used), live reporting is not supported. Impala-shell will disable live_progress if the mode is detected. Testing: - Added and updated tests in test_shell_interactive.py and test_shell_commandline.py - Successfully ran all shell related tests is detected. Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 --- M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py 5 files changed, 33 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/15219/3 -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 3 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15219/2/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/15219/2/tests/shell/test_shell_interactive.py@535 PS2, Line 535: " flake8: E501 line too long (91 > 90 characters) -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 2 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 04:59:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by enabling live progress in interactive mode
Hello Andrew Sherman, David Knupp, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15219 to look at the new patch set (#2). Change subject: IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode .. IMPALA-9384: Improve Impala shell usability by enabling live_progress in interactive mode In order to improve usability, this patch would like to make Impala shell show query processing status by default. The patch enables shell option live_progress when a user launches impala shell in the interactive mode. The patch also provides a new command line flag “--disable_live_progress”, which allows a user to disable live_progress at runtime. In the interactive mode, a user can disable live_progress by either using the proposed command line flag or setting the option as False in the config file. As for in the non-interactive mode (when the -q or -f options are used), live reporting is not supported. Impala-shell will disable live_progress if the mode is detected. Testing: - Added and updated tests in test_shell_interactive.py and test_shell_commandline.py - Successfully ran all shell related tests Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 --- M shell/impala_shell.py M shell/impala_shell_config_defaults.py M shell/option_parser.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py 5 files changed, 33 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/15219/2 -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 2 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5396/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 03:05:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Hello Anonymous Coward (572), Tim Armstrong, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15334 to look at the new patch set (#2). Change subject: IMPALA-9400: Initial Ozone support .. IMPALA-9400: Initial Ozone support Ozone is a block-based filesystem store and is similar to HDFS in many aspects. This patch adds initial support for reading data stored on Ozone and writing data to Ozone. It adds Ozone to the whitelist of Impala writable filesystems and adds the necessary Ozone client jars to the Impala classpath. Initially, Ozone will be treated as a remote storage service, similar to how Impala treats S3 and ABFS. Ozone will typically be deployed on a separate storage cluster, so all reads will be remote. Rather than updating to a new CDP GBN with the added Ozone client jars, I just modified the existing GBN to include the new jars in the Maven repo. Testing: * Manually tested that connecting to an external Ozone cluster works and that Impala can create an Ozone-backed table, write to the table, read from the table, and drop the table * Ran core tests with and without USE_CDP_HIVE=true Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 --- M be/src/exec/hdfs-table-sink.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M impala-parent/pom.xml 9 files changed, 72 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/2 -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: > I think this is fine to leave enabled by default. This would be an argument Sounds good. Will follow up on the EC support separately. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 02:20:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] santizer and -release flags should cause buildall.sh to exit
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15341 ) Change subject: santizer and -release flags should cause buildall.sh to exit .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e Gerrit-Change-Number: 15341 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 01:53:03 + Gerrit-HasComments: No
[Impala-ASF-CR] santizer and -release flags should cause buildall.sh to exit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15341 ) Change subject: santizer and -release flags should cause buildall.sh to exit .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5395/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e Gerrit-Change-Number: 15341 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 01:29:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5394/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 01:23:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5441/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 01:14:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 01:14:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 01:14:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5393/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 01:13:07 + Gerrit-HasComments: No
[Impala-ASF-CR] santizer and -release flags should cause buildall.sh to exit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15341 ) Change subject: santizer and -release flags should cause buildall.sh to exit .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5440/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e Gerrit-Change-Number: 15341 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 01:07:15 + Gerrit-HasComments: No
[Impala-ASF-CR] santizer and -release flags should cause buildall.sh to exit
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15341 Change subject: santizer and -release flags should cause buildall.sh to exit .. santizer and -release flags should cause buildall.sh to exit Currently, if any sanitizer (or clang-tidy) flag is added to buildall.sh and the -release flag is added, then buildall.sh will silently ignore the -release flag. Impala does not support adding sanitizer flags to debug/release builds. Sanitizers, release, and debug builds are all distinct and use their own set of compile flags. This patch changes the behavior of buildall.sh so that if -release and any sanitizer flag is specified, the build exits with the error: "ERROR: more than one CMake build type defined: RELEASE TSAN" Testing: * './buildall.sh -skiptests -noclean -tsan -release' fails (as expected) * './buildall.sh -skiptests -noclean -tsan' passes * './buildall.sh -notests -noclean -codecoverage -release' passes Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e --- M buildall.sh 1 file changed, 10 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/15341/1 -- To view, visit http://gerrit.cloudera.org:8080/15341 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ide0c2017d4e5abbf6fcb25c890d241bbcee8422e Gerrit-Change-Number: 15341 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15339 ) Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5392/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 03 Mar 2020 00:59:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9369: Make createInsertEvents() async.
Vihang Karajgaonkar has posted comments on this change. (
http://gerrit.cloudera.org:8080/15263 )
Change subject: IMPALA-9369: Make createInsertEvents() async.
..
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15263/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/15263/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4393
PS2, Line 4393: fireInsertEventsAsync(table.getDb().getName(),
This implementation will spin up a singleThreadExecutor for each partition
which was inserted which might be expensive. Can we collect all the insert
events first and then call the async method so all the insert events are
processed by same single thread?
May be something like:
CompletableFuture.runAsync(() -> {
try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
for (InsertEventInfo info : insertEventInfos) {
MetaStoreUtil.fireInsertEvent(info);
} catch (Exception e) {
LOG.error(...);
}},
Executors.newSingleThreadExecutor()).thenRun(()->fireInsertEventThread.shutdown)
})
--
To view, visit http://gerrit.cloudera.org:8080/15263
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I97802a5c03abc067fccf9e3a9d0047324626706e
Gerrit-Change-Number: 15263
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada
Gerrit-Reviewer: Anurag Mantripragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Comment-Date: Tue, 03 Mar 2020 00:52:01 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5438/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 00:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15318 ) Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. Patch Set 4: Code-Review+2 rebased and carrying forward Tim and Csaba's +2 -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 00:40:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Hello Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15318 to look at the new patch set (#4). Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created Currently the exprs created in plan nodes and sink configs were being closed in their respective exec nodes and data sink instances (that were created using them) when the fragment instance closes. This patch ensure that they are closed from the plan nodes and sink configs themselves so that when these are shared among instances, they don't prematurely close the expressions. Testing: Successfully passed exhaustive tests. Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/scan-node.cc M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M be/src/runtime/data-stream-test.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h 41 files changed, 225 insertions(+), 139 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15318/4 -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/15318 )
Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and
sink configs where they were created
..
Patch Set 3:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/aggregation-node-base.cc
File be/src/exec/aggregation-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/aggregation-node-base.cc@50
PS2, Line 50: void AggregationPlanNode::Close() {
: for (AggregatorConfig* config : aggs_) config->Close();
: PlanNode::Close();
: }
> I don't want to hold this up, just wanted to mention that the whole Expr/Pl
I agree, that will save us the effort of adding the close methods, but will
have to investigate more to figure out if decoupling of cache_entry_ management
and exprs is cleaner and intuitive. will add a JIRA for this:IMPALA-9454
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/exec-node.h@73
PS2, Line 73: /// Close() releases all resources that were allocated in
Init().
> The second sentence is probably at risk of getting stale, maybe remove it?
Done. Here and in data-sink.h
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/exec-node.cc
File be/src/exec/exec-node.cc:
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/exec-node.cc@91
PS2, Line 91: for (auto& child : children_) {
> Consider using ranged for, just for consistency?
Done
--
To view, visit http://gerrit.cloudera.org:8080/15318
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d
Gerrit-Change-Number: 15318
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Tue, 03 Mar 2020 00:30:34 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Hello Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15318 to look at the new patch set (#3). Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created .. IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created Currently the exprs created in plan nodes and sink configs were being closed in their respective exec nodes and data sink instances (that were created using them) when the fragment instance closes. This patch ensure that they are closed from the plan nodes and sink configs themselves so that when these are shared among instances, they don't prematurely close the expressions. Testing: Successfully passed exhaustive tests. Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d --- M be/src/exec/aggregation-node-base.cc M be/src/exec/aggregation-node-base.h M be/src/exec/aggregator.cc M be/src/exec/aggregator.h M be/src/exec/analytic-eval-node.cc M be/src/exec/analytic-eval-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator.cc M be/src/exec/grouping-aggregator.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node-base.h M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/exec/nested-loop-join-node.cc M be/src/exec/nested-loop-join-node.h M be/src/exec/non-grouping-aggregator.h M be/src/exec/partial-sort-node.cc M be/src/exec/partial-sort-node.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/exec/scan-node.cc M be/src/exec/sort-node.cc M be/src/exec/sort-node.h M be/src/exec/topn-node.cc M be/src/exec/topn-node.h M be/src/exec/union-node.cc M be/src/exec/union-node.h M be/src/exec/unnest-node.cc M be/src/exec/unnest-node.h M be/src/runtime/data-stream-test.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h 41 files changed, 224 insertions(+), 141 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/15318/3 -- To view, visit http://gerrit.cloudera.org:8080/15318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d Gerrit-Change-Number: 15318 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 00:26:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes This patch moves the code responsible for codegening from partitioned hash join's exec node to plan node and from all data sinks to their respective config classes. Testing: Manually verified that codegen works for all modified exec nodes and data sinks. Ran all exhaustive tests successfully. Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Reviewed-on: http://gerrit.cloudera.org:8080/15322 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/grouping-aggregator.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h 16 files changed, 281 insertions(+), 190 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 4 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 03 Mar 2020 00:22:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. IMPALA-9444: Fix URL for postgresql jar download The current URL uses http://central.maven.org, which has been decommissioned as part of the transition to HTTPS. See: https://central.sonatype.org/articles/2019/Jul/15/central-http-deprecation-update/ https://central.sonatype.org/articles/2020/Jan/15/501-https-required-error/ This switches the URL to use https://repo.maven.apache.org/. Testing: - Removed postgresql jar, ran bin/create-test-configuration.sh, verified that it downloaded the jar. Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Reviewed-on: http://gerrit.cloudera.org:8080/15337 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins --- M bin/create-test-configuration.sh 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Joe McDonnell: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 3 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9452: slightly reduce reservation for test spilling aggs
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15339 Change subject: IMPALA-9452: slightly reduce reservation for test_spilling_aggs .. IMPALA-9452: slightly reduce reservation for test_spilling_aggs As far as I can tell, the query failed to spill because the pre-agg was able to release reservation before the post-agg needed it. Probably there is some variance because of buffering in the exchange. This change slightly reduces the reservation to minimise the chance of this recurring. Also remove a duplicated instance of this test. Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 --- M testdata/workloads/functional-query/queries/QueryTest/spilling-aggs.test 1 file changed, 1 insertion(+), 24 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/15339/1 -- To view, visit http://gerrit.cloudera.org:8080/15339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ifb8376e2e12d3f73d6c0e27c697be4fc86f9c755 Gerrit-Change-Number: 15339 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-8690: Add LIRS cache eviction algorithm
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15306 ) Change subject: IMPALA-8690: Add LIRS cache eviction algorithm .. Patch Set 9: (9 comments) http://gerrit.cloudera.org:8080/#/c/15306/9//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15306/9//COMMIT_MSG@17 PS9, Line 17: the the startup flag http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/cache.cc File be/src/util/cache/cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/cache.cc@87 PS9, Line 87: case Cache::EvictionPolicy::FIFO: nit: formatting http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc File be/src/util/cache/lirs-cache.cc: http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@42 PS9, Line 42: DEFINE_double_hidden(lirs_tombstone_multiple, 2.00, I don't see anywhere you validate that this value is reasonable. Might be good to do a quick check somewhere and log a warning or something to help avoid people accidentally mis-configuring this and getting bad perf as a result. http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@89 PS9, Line 89: // oldest PROTECTED entry is demoted to UNPROTECTED. Might be nice to also include the semantics of UNINIT in this paragraph. http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@93 PS9, Line 93: last entry I think some of the comments around the ordering of the lists get a little confusing - if I understand correctly, by 'last entry' here you mean the oldest entry, which is actually at the front of the queue. Similarly, later in this paragraph you refer to the elements being removed as being 'at the end' but TrimRecencyQueue refers to them as being at the 'front of the queue'. Would be good to settle on a small number of terms, eg. oldest/front and newest/back, and use them consistently everywhere. This is a little nit-pick-ier, but it might also be nice to be consistent about whether they're referred to as 'list' or 'queue' http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@100 PS9, Line 100: cache list http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@241 PS9, Line 241: DCHECK_LT(cur_state.ref_count, std::numeric_limits::max()); Is there anything that guarantees this won't be the case, i.e. as written why wouldn't it be valid to call Lookup() a bunch of times on the same element and violate this? Admittedly it seems unlikely this would be violated in a real workload, but we may still need to handle it. http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@242 PS9, Line 242: ++ nit: its impala convention to always prefer this at the beginning, i.e. '++cur_state.ref_count' http://gerrit.cloudera.org:8080/#/c/15306/9/be/src/util/cache/lirs-cache.cc@466 PS9, Line 466: MoveToQueueBack MoveToRecencyQueueBack -- To view, visit http://gerrit.cloudera.org:8080/15306 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I670fa4b2b7c93998130dc4e8b2546bb93e9a84f8 Gerrit-Change-Number: 15306 Gerrit-PatchSet: 9 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Mon, 02 Mar 2020 23:28:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15335 ) Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py This makes the --scale option mandatory when running ./bin/single_node_perf_run.py. If the option is not set, the script attempts to run the workloads against the database '[workload-name]None_[file-format]', which is typically not what the user wants. Makes some minor documentation improvements to the script. Testing: * Confirmed that running without the --scale option set causes the script to error out with a help message Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Reviewed-on: http://gerrit.cloudera.org:8080/15335 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M bin/single_node_perf_run.py 1 file changed, 13 insertions(+), 2 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15335 ) Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 22:35:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3343: Make impala-shell compatible with python 3.
David Knupp has posted comments on this change. (
http://gerrit.cloudera.org:8080/15132 )
Change subject: IMPALA-3343: Make impala-shell compatible with python 3.
..
Patch Set 17:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/15132/15/tests/shell/test_shell_commandline.py@485
PS15, Line 485: if SHELL_IS_PYTHON_2:
> I got confused by the scenario, but I checked this out and played around an
Still digging into this. The proposed change of cname.decode('utf8') fails
under python 3 because with python 3:
'str' object has no attribute 'decode'
In the meantime, with that change, the test against the built tarball still
fails:
E Query: select '?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,?, ?,
?, ?, ?, ?,?, ?, ?, ?, ?, ?, ?, ?, ?\t'
E Query submitted at: 2020-03-02 14:06:17 (Coordinator:
http://dknupp-desktop:25000)
E Query progress can be monitored at:
http://dknupp-desktop:25000/query_plan?query_id=c649f690ad99d702:8571f574
E Unknown Exception : 'ascii' codec can't encode character u'\u0430' in
position 107: ordinal not in range(128)
E Traceback (most recent call last):
E File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
line 1934, in
E impala_shell_main()
E File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
line 1888, in impala_shell_main
E if execute_queries_non_interactive_mode(options, query_options):
E File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
line 1697, in execute_queries_non_interactive_mode
E shell.execute_query_list(queries))
E File
"/home/dknupp/Impala/shell/build/impala-shell-3.4.0-SNAPSHOT/impala_shell.py",
line 1531, in execute_query_list
E print('Could not execute command: %s' % q, file=sys.stderr)
E UnicodeEncodeError: 'ascii' codec can't encode character u'\u0410' in
position 35: ordinal not in range(128)
--
To view, visit http://gerrit.cloudera.org:8080/15132
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb75e162bac0faeae3e12106c15da39cbfb8b462
Gerrit-Change-Number: 15132
Gerrit-PatchSet: 17
Gerrit-Owner: David Knupp
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 02 Mar 2020 22:09:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5391/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 21:59:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 2: Tested manually, skipping CI to unblock builds faster. -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 21:31:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. IMPALA-9445: skip test_local_tz_conversion for hbase This skips the test for filesystems where we don't start HBase. It's a bit unfortunate that we have to disable all of this test, so this is a stopgap to get builds green. Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Reviewed-on: http://gerrit.cloudera.org:8080/15338 Reviewed-by: Csaba Ringhofer Tested-by: Tim Armstrong --- M tests/custom_cluster/test_local_tz_conversion.py 1 file changed, 4 insertions(+), 1 deletion(-) Approvals: Csaba Ringhofer: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15338/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15338/2//COMMIT_MSG@10 PS2, Line 10: bit unfortunate I think that it doesn't add extra coverage to run this on S3, so it's ok to leave it this way. I added this test only to avoid the very specific bug of depending on flag use_local_tz_for_unix_timestamp_conversions when reading timestamps from different fileformats, as this bug could not be detected by tests that run with default flags. -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 21:31:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 21:31:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 02 Mar 2020 21:23:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Hello Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15338 to look at the new patch set (#2). Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. IMPALA-9445: skip test_local_tz_conversion for hbase This skips the test for filesystems where we don't start HBase. It's a bit unfortunate that we have to disable all of this test, so this is a stopgap to get builds green. Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d --- M tests/custom_cluster/test_local_tz_conversion.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/15338/2 -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5389/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 20:15:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5390/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 02 Mar 2020 20:11:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9424: Add six to shell/ext-py
Thomas Tauber-Marshall has abandoned this change. ( http://gerrit.cloudera.org:8080/15294 ) Change subject: IMPALA-9424: Add six to shell/ext-py .. Abandoned -- To view, visit http://gerrit.cloudera.org:8080/15294 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: If5567a28884b75643929f64f0ca913fc7ce42e5d Gerrit-Change-Number: 15294 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5388/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:44:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15338 ) Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15338/1/tests/custom_cluster/test_local_tz_conversion.py File tests/custom_cluster/test_local_tz_conversion.py: http://gerrit.cloudera.org:8080/#/c/15338/1/tests/custom_cluster/test_local_tz_conversion.py@22 PS1, Line 22: from tests.common.skip import SkipIfBuildType, SkipIfABFS, SkipIfADLS, SkipIfS3 flake8: F401 'tests.common.skip.SkipIfBuildType' imported but unused -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Mon, 02 Mar 2020 19:38:21 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9445: skip test local tz conversion for hbase
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15338 Change subject: IMPALA-9445: skip test_local_tz_conversion for hbase .. IMPALA-9445: skip test_local_tz_conversion for hbase This skips the test for filesystems where we don't start HBase. It's a bit unfortunate that we have to disable all of this test, so this is a stopgap to get builds green. Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d --- M tests/custom_cluster/test_local_tz_conversion.py 1 file changed, 4 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/38/15338/1 -- To view, visit http://gerrit.cloudera.org:8080/15338 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9fb603ebe0a5820d2eed0b6ec5dca2980fd5521d Gerrit-Change-Number: 15338 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 2: Code-Review+2 carrying over Tim's +2 -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:32:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5437/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:33:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:33:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/15322 ) Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/15322/1/be/src/exec/partitioned-hash-join-builder.cc File be/src/exec/partitioned-hash-join-builder.cc: http://gerrit.cloudera.org:8080/#/c/15322/1/be/src/exec/partitioned-hash-join-builder.cc@1162 PS1, Line 1162: codegen, true, *hash_table_config_, _build_row_fn)); > line too long (92 > 90) Done -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:32:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes
Hello Daniel Becker, Tim Armstrong, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15322 to look at the new patch set (#2). Change subject: IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes .. IMPALA-4080 [part 6]: Move codegen from join's exec node to plan node and from data sinks to their config classes This patch moves the code responsible for codegening from partitioned hash join's exec node to plan node and from all data sinks to their respective config classes. Testing: Manually verified that codegen works for all modified exec nodes and data sinks. Ran all exhaustive tests successfully. Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a --- M be/src/exec/blocking-join-node.cc M be/src/exec/blocking-join-node.h M be/src/exec/data-sink.cc M be/src/exec/data-sink.h M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/grouping-aggregator.cc M be/src/exec/nested-loop-join-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/partitioned-hash-join-node.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/krpc-data-stream-sender.cc M be/src/runtime/krpc-data-stream-sender.h M be/src/runtime/runtime-state.h 16 files changed, 281 insertions(+), 190 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/15322/2 -- To view, visit http://gerrit.cloudera.org:8080/15322 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8fe54b4040d6e3c2fc224f7cd6c8c0bfe813707a Gerrit-Change-Number: 15322 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5436/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:28:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. Patch Set 2: Code-Review+2 (1 comment) Carry +2 http://gerrit.cloudera.org:8080/#/c/15337/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15337/1//COMMIT_MSG@11 PS1, Line 11: https://central.sonatype.org/articles/2019/Jul/15/central-http-deprecation-update/ > Add https://central.sonatype.org/articles/2020/Jan/15/501-https-required-er Added some context -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:27:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15337 to look at the new patch set (#2). Change subject: IMPALA-9444: Fix URL for postgresql jar download .. IMPALA-9444: Fix URL for postgresql jar download The current URL uses http://central.maven.org, which has been decommissioned as part of the transition to HTTPS. See: https://central.sonatype.org/articles/2019/Jul/15/central-http-deprecation-update/ https://central.sonatype.org/articles/2020/Jan/15/501-https-required-error/ This switches the URL to use https://repo.maven.apache.org/. Testing: - Removed postgresql jar, ran bin/create-test-configuration.sh, verified that it downloaded the jar. Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d --- M bin/create-test-configuration.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15337/2 -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15337 ) Change subject: IMPALA-9444: Fix URL for postgresql jar download .. Patch Set 1: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/15337/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15337/1//COMMIT_MSG@11 PS1, Line 11: Add https://central.sonatype.org/articles/2020/Jan/15/501-https-required-error/ for context, maybe? -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:20:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: > Do you mean add the flag ALLOW_OZONE_FILES and set it to false by default, I think this is fine to leave enabled by default. This would be an argument for enabling erasure encoding. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 19:15:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9444: Fix URL for postgresql jar download
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15337 Change subject: IMPALA-9444: Fix URL for postgresql jar download .. IMPALA-9444: Fix URL for postgresql jar download The current URL uses central.maven.org, which has been decommissioned. This switches the URL to use https://repo.maven.apache.org/. Testing: - Removed postgresql jar, ran bin/create-test-configuration.sh, verified that it downloaded the jar. Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d --- M bin/create-test-configuration.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/37/15337/1 -- To view, visit http://gerrit.cloudera.org:8080/15337 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I7ee9a1ce77bc3f8c6b3f728633cafe4eb37e669d Gerrit-Change-Number: 15337 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-9425 (part 1): Introduce uuids for impalads
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15321 ) Change subject: IMPALA-9425 (part 1): Introduce uuids for impalads .. Patch Set 2: (5 comments) http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-http-handler.cc@498 PS2, Line 498: membership_snapshot nit: DCHECK this is not null? http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-http-handler.cc@503 PS2, Line 503: backend_id Tim already mentioned it, but might be good to expose the UUID here; it looks like its possible for there to be two separate entries in the table, with the same address, so adding the uuid would help differentiate duplicate addresses http://gerrit.cloudera.org:8080/#/c/15321/2/common/thrift/StatestoreService.thrift File common/thrift/StatestoreService.thrift: http://gerrit.cloudera.org:8080/#/c/15321/2/common/thrift/StatestoreService.thrift@62 PS2, Line 62: 1: required Types.BackendId backend_id; nit: add docs http://gerrit.cloudera.org:8080/#/c/15321/2/tests/common/impala_test_suite.py File tests/common/impala_test_suite.py: http://gerrit.cloudera.org:8080/#/c/15321/2/tests/common/impala_test_suite.py@1080 PS2, Line 1080: def wait_for_state(self, handle, expected_state, timeout, client=None): : """Waits for the given 'query_handle' to reach the 'expected_state'. If it does not : reach the given state within 'timeout' seconds, the method throws an AssertionError. : """ : self.wait_for_any_state(handle, [expected_state], timeout, client) : : def wait_for_any_state(self, handle, expected_states, timeout, client=None): : """Waits for the given 'query_handle' to reach one of 'expected_states'. If it does : not reach one of the given states within 'timeout' seconds, the method throws an : AssertionError. Returns the final state. update docs for 'client' param http://gerrit.cloudera.org:8080/#/c/15321/2/tests/custom_cluster/test_restart_services.py File tests/custom_cluster/test_restart_services.py: http://gerrit.cloudera.org:8080/#/c/15321/2/tests/custom_cluster/test_restart_services.py@103 PS2, Line 103: # Debug action to delay statestore updates to give the restarted impalad time to : # register itself before a membership topic update is generated. : self._start_impala_cluster( : ["--state_store_args=--debug_actions=DO_SUBSCRIBER_UPDATE:JITTER@1"], : num_coordinators=1, cluster_size=3) : assert len(self.cluster.impalads) == 3 nit: use CustomClusterTestSuite.with_args? -- To view, visit http://gerrit.cloudera.org:8080/15321 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf8067349ed6b765f6fed830b7140f60738e9061 Gerrit-Change-Number: 15321 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 18:53:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9425 (part 1): Introduce uuids for impalads
Fang-Yu Rao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15321 ) Change subject: IMPALA-9425 (part 1): Introduce uuids for impalads .. Patch Set 2: (4 comments) Hi Thomas, I have read this patch but do not have any major comment. I raised those questions mostly because I am not familiar with this part of our BE codebase and would like to check if my understanding is correct. Thank you very much! http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/scheduling/cluster-membership-mgr.cc File be/src/scheduling/cluster-membership-mgr.cc: http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/scheduling/cluster-membership-mgr.cc@135 PS2, Line 135: bool ss_is_recovering = statestore_subscriber_ != nullptr It is not directly related to this patch. But is it ok for us to move this following statement to the place where we have known there is at least one condition in the following if-statement at Line 148 that evaluates to false? I just like to know whether or not my understanding is correct. Thanks! http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/scheduling/cluster-membership-mgr.cc@347 PS2, Line 347: // Send notifications to all callbacks registered to receive updates. Is it true that the callbacks referred to here correspond to the input argument to the function cluster_membership_mgr_->RegisterUpdateCallbackFn() at Line 548 in exec-env.cc? I'd just like to check if my understanding is correct. http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-http-handler.cc File be/src/service/impala-http-handler.cc: http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-http-handler.cc@506 PS2, Line 506: Value location_name(TNetworkAddressToString(be_entry.second.address).c_str(), 'be_entry.second' is of the type TBackendDescriptor, which has 3 fields that are of the type TNetworkAddress, i.e., 'address', 'debug_http_address', and 'krpc_address'. Is it okay to use any of the other two instead, i.e., 'debug_http_address' or 'krpc_address'? Thanks! http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/15321/2/be/src/service/impala-server.cc@1723 PS2, Line 1723: map> queries_to_cancel; Is it possible to use 'backend_id' instead of 'address' to construct the map? Or the change would be too huge since the elements in 'queries_to_cancel' will also be used in other function later on, e.g., cancellation_thread_pool_->Offer(CancellationWork::BackendFailure(cancellation_entry.first, cancellation_entry.second));"? -- To view, visit http://gerrit.cloudera.org:8080/15321 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icf8067349ed6b765f6fed830b7140f60738e9061 Gerrit-Change-Number: 15321 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 18:52:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: > I don't want to scope creep this, but another thought I had was that it see Do you mean add the flag ALLOW_OZONE_FILES and set it to false by default, or do you mean set ALLOW_ERASURE_CODED_FILES=true by default? -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 18:38:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: > From what I know so far, Ozone does expose block locations. I don't want to scope creep this, but another thought I had was that it seems weird to allow Ozone by default but not allow HDFS erasure coding (ALLOW_ERASURE_CODED_FILES=false). Maybe we should just leave as-is though... might be worth thinking about flipping the result. Treating it as a remote FS by default makes sense to me, I think, since we then avoid the overhead of fetching and storing the block locations. No we don't support rack-aware scheduling in Impala.. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 18:27:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Anonymous Coward (572) has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: Ozone does support block locations (LocatedFIleStatus) and we have tested it against Hive. It does not however support short circuit reads. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Anonymous Coward (572) Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 18:03:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9384: Improve Impala shell usability by automatically enable live progress in the interactive mode
David Knupp has posted comments on this change. ( http://gerrit.cloudera.org:8080/15219 ) Change subject: IMPALA-9384: Improve Impala shell usability by automatically enable live_progress in the interactive mode .. Patch Set 1: > Patch Set 1: > David and Andrew, please let me know if you agree with option 2? Thanks! Thanks Alice -- Andrew and I both in favor of option 2. -- To view, visit http://gerrit.cloudera.org:8080/15219 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3765b775f663fa227e59728acffe4d5ea9a5e2d3 Gerrit-Change-Number: 15219 Gerrit-PatchSet: 1 Gerrit-Owner: Alice Fan Gerrit-Reviewer: Alice Fan Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:42:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15335 ) Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5435/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:38:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: > I have some questions about the block locations and scheduling. Does this l >From what I know so far, Ozone does expose block locations. This patch changes supportsStorageIds so that it returns true for Ozone filesystems as well (this should make the HdfsScanNode fetch the block locations and perform scheduling based on the block locations). However, I think I might revert that part of the patch. I believe, for the most part, we expect to deploy Ozone on a dedicated storage cluster that is separate from the Impala compute cluster - e.g. separate compute vs storage cluster. So I think, at least for now, its okay if we treat Ozone as remote storage (similar to what we do for ABFS and S3A). Even with separate storage + compute clusters, block locations might help with rack-aware scheduling, but I don't think we support that in Impala yet? I'll make the fixes for the DiskIoMgr. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:37:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15335 ) Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5387/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:21:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15332 ) Change subject: IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5386/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c Gerrit-Change-Number: 15332 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 17:08:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15332 ) Change subject: IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5385/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c Gerrit-Change-Number: 15332 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 17:07:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15335 ) Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:03:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15334/1//COMMIT_MSG@18 PS1, Line 18: I have some questions about the block locations and scheduling. Does this load block locations and do locality-based scheduling? It looks like there isn't any code added to DiskIoMgr to route the reads to a particular queue - it seems like in DiskIoMgr::AssignQueue() it falls through in a weird way. OK if we want to defer this but should probably file a JIRA. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 17:02:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4080 [part 3]: Moved codegen code for TupleRowComparator into a Config object
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15281 ) Change subject: IMPALA-4080 [part 3]: Moved codegen code for TupleRowComparator into a Config object .. Patch Set 3: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/15281/3/be/src/util/tuple-row-compare.h File be/src/util/tuple-row-compare.h: http://gerrit.cloudera.org:8080/#/c/15281/3/be/src/util/tuple-row-compare.h@164 PS3, Line 164: std::vector nulls_first_; Could be const ref as the bool->int8_t conversion is done in TupleRowComparatorConfig -- To view, visit http://gerrit.cloudera.org:8080/15281 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I265604ecbae2c80ff59e697fddfeea4976b25cd9 Gerrit-Change-Number: 15281 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 16:56:47 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9439: Make --scale a mandatory option in single node perf run.py
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15335 Change subject: IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py .. IMPALA-9439: Make --scale a mandatory option in single_node_perf_run.py This makes the --scale option mandatory when running ./bin/single_node_perf_run.py. If the option is not set, the script attempts to run the workloads against the database '[workload-name]None_[file-format]', which is typically not what the user wants. Makes some minor documentation improvements to the script. Testing: * Confirmed that running without the --scale option set causes the script to error out with a help message Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b --- M bin/single_node_perf_run.py 1 file changed, 13 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/15335/1 -- To view, visit http://gerrit.cloudera.org:8080/15335 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9ad13580f8f74388981a37d6960087d95cde574b Gerrit-Change-Number: 15335 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and sink configs where they were created
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/15318 )
Change subject: IMPALA-4080 [part 5]: Invoke close on Exprs from plan nodes and
sink configs where they were created
..
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/aggregation-node-base.cc
File be/src/exec/aggregation-node-base.cc:
http://gerrit.cloudera.org:8080/#/c/15318/2/be/src/exec/aggregation-node-base.cc@50
PS2, Line 50: void AggregationPlanNode::Close() {
: for (AggregatorConfig* config : aggs_) config->Close();
: PlanNode::Close();
: }
I don't want to hold this up, just wanted to mention that the whole
Expr/PlanNode closing logic looks unnecessary to me: if I didn't miss
something, then the only resources that need closing are 'cache_entry_'s that
are used by ScalarFnCall to get UDF/UDA files. These could be registered to
the fragment instance when we get the file and released during closing.
--
To view, visit http://gerrit.cloudera.org:8080/15318
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7b79d8b8f8e740609676dd13bd300a5514c65b0d
Gerrit-Change-Number: 15318
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 02 Mar 2020 16:24:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY
Adam Tamas has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/15332 ) Change subject: IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY .. IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY Modified the sql_parser.cup to accept the reversed syntax. Testing: -Added an extra AnalyzesOk into ‘AnalyzeKuduDDLTest()’ -> ‘testDDlsOnKuduTable()’ in AnalyzeKuduDDLTest.java where the RANGE() and HASH() pair is swapped. Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c --- M fe/src/main/cup/sql-parser.cup M fe/src/test/java/org/apache/impala/analysis/AnalyzeKuduDDLTest.java 2 files changed, 10 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/15332/2 -- To view, visit http://gerrit.cloudera.org:8080/15332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c Gerrit-Change-Number: 15332 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY
Adam Tamas has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15332 Change subject: IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY .. IMPALA-7686: Allow RANGE() clause before HASH() clause for PARTITION BY Modified the sql_parser.cup to accept the reversed syntax. Testing: -swapped an instance of RANGE() + HASH() pair in test_kudu.py -> ‘test_primary_key_and_distribution()’ and left the rest to check if it is still create the right table with ‘assert_show_create_equals()’. Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c --- M fe/src/main/cup/sql-parser.cup M tests/query_test/test_kudu.py 2 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/15332/1 -- To view, visit http://gerrit.cloudera.org:8080/15332 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I914e340e9acfb0f49c7d4f78705dbd9bde0aec8c Gerrit-Change-Number: 15332 Gerrit-PatchSet: 1 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5384/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 16:11:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15334 ) Change subject: IMPALA-9400: Initial Ozone support .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5383/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 02 Mar 2020 15:54:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 8: Code-Review+2 (4 comments) http://gerrit.cloudera.org:8080/#/c/15104/5//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/15104/5//COMMIT_MSG@20 PS5, Line 20: Note, typo: testing http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-columnar-scanner-ir.cc File be/src/exec/hdfs-columnar-scanner-ir.cc: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-columnar-scanner-ir.cc@23 PS5, Line 23: scratch_batch_ nit: the != nullptr in written out in most of Impala http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-orc-scanner.cc File be/src/exec/hdfs-orc-scanner.cc: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-orc-scanner.cc@702 PS5, Line 702: nit: the != nullptr in written out in most of Impala http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.cc File be/src/exec/orc-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.cc@264 PS5, Line 264: >col_id_path_map_[ Can you mention it in the commit message that no scratch batch is used if there are collections? This will lead to also not having codegen in this case if the we want to use the same functions as Parquet, which always uses scratch batches. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 15:30:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/15104/7/be/src/exec/hdfs-scanner.h File be/src/exec/hdfs-scanner.h: http://gerrit.cloudera.org:8080/#/c/15104/7/be/src/exec/hdfs-scanner.h@176 PS7, Line 176: Not inlined in IR so it can be replaced with a constant. > nit: this comment is applicable to both 'tuple_byte_size' functions, so it Done -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 15:26:44 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15104 to look at the new patch set (#8). Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. IMPALA-9228: ORC scanner reads rows into scratch batch Because of performance considerations this change enhances ORC scanner to populate a scratch batch on a column-by-column manner using data from the column readers. Once this is done the parquet code was reused to apply runtime filter and conjuncts and to populate the outgoing row batch. This approach reduces the number of virtual function calls and takes advantage of the columnar orientation of the data to enhance scan performance. Additionally, introducing the scratch batch concept also opens the door for codegen runtime filtering and applying conjuncts. Note, this change doesn't cover collection types just primitive types and struct. Tesing: - Re-run the full test suite to verify that no regression is introduced. - Checked the performance impact by running TPCH workload on a scale 25 database using single_node_perf_run.py. The total query runtime is decreased by 0-20% depending on how scan heavy the particular query was. The more scan heavy the query is the more performance gain I observe. Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/CMakeLists.txt R be/src/exec/hdfs-columnar-scanner-ir.cc A be/src/exec/hdfs-columnar-scanner.cc A be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h R be/src/exec/scratch-tuple-batch.h 15 files changed, 425 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/15104/8 -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 8 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-9400: Initial Ozone support
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15334 Change subject: IMPALA-9400: Initial Ozone support .. IMPALA-9400: Initial Ozone support Ozone is a block-based filesystem store and is similar to HDFS in many aspects. This patch adds initial support for reading data stored on Ozone and writing data to Ozone. It adds Ozone to the whitelist of Impala writable filesystems and adds the necessary Ozone client jars to the Impala classpath. Rather than updating to a new CDP GBN with the added Ozone client jars, I just modified the existing GBN to include the new jars in the Maven repo. Testing: * Manually tested that connecting to an external Ozone cluster works and that Impala can create an Ozone-backed table, write to the table, read from the table, and drop the table * Ran core tests with and without USE_CDP_HIVE=true Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 --- M be/src/exec/hdfs-table-sink.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M impala-parent/pom.xml 7 files changed, 55 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/15334/1 -- To view, visit http://gerrit.cloudera.org:8080/15334 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I2544ff61e8767f061253a62a01ce2b7dfc384a99 Gerrit-Change-Number: 15334 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5382/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 7 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 14:34:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5381/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 13:58:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Adam Tamas has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. Patch Set 7: (1 comment) http://gerrit.cloudera.org:8080/#/c/15288/6/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/15288/6/tests/webserver/test_web_pages.py@423 PS6, Line 423: > nit: This should be in the same line as the string, also the second line sh Done -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 7 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 13:48:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5380/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 6 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 13:48:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Adam Tamas has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. IMPALA-6360: Don't show full query statement on Impala webUI by default Added the ‘query_stmt_size’ flag to impala-server.cc with default value of 250 and modified the ‘ImpalaHttpHandler::QueryStateToJson()’ to truncate the end of the statements if they are too long. Testing: -manual testing in the WebUI. -added 'test_query_stmt()' to test_web_pages.py Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 --- M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M tests/webserver/test_web_pages.py 3 files changed, 32 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15288/7 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 7 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/5379/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 13:41:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Tamas Mate has posted comments on this change. ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. Patch Set 6: Code-Review+1 (1 comment) Thanks for the updates. Just found one more nit, other than that LGTM! http://gerrit.cloudera.org:8080/#/c/15288/6/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/15288/6/tests/webserver/test_web_pages.py@423 PS6, Line 423: """ nit: This should be in the same line as the string, also the second line should start 3 characters earlier. Some words might also fit in the first line till col 90. -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 6 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate Gerrit-Comment-Date: Mon, 02 Mar 2020 13:40:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 7: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/15104/7/be/src/exec/hdfs-scanner.h File be/src/exec/hdfs-scanner.h: http://gerrit.cloudera.org:8080/#/c/15104/7/be/src/exec/hdfs-scanner.h@176 PS7, Line 176: Not inlined in IR so it can be replaced with a constant. nit: this comment is applicable to both 'tuple_byte_size' functions, so it should be put before L174, similarly to Base. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 13:35:23 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 7: (3 comments) http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-scanner.h File be/src/exec/hdfs-scanner.h: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/hdfs-scanner.h@a566 PS5, Line 566: : : : : > nit: maybe these could be moved together to public. Done http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.h File be/src/exec/orc-column-readers.h: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.h@188 PS5, Line 188: Status ReadValue(int row_idx, Tuple* tuple, MemPool* pool) final WARN_UNUSED_RESULT; > Since ReadValue() is public I think you don't need the friend declarations Friend declarations are still needed for accessing 'derived->batch_' from OrcPrimitiveColumnReader::ReadValueBatch() http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.cc File be/src/exec/orc-column-readers.cc: http://gerrit.cloudera.org:8080/#/c/15104/5/be/src/exec/orc-column-readers.cc@221 PS5, Line 221: ErrorMsg msg(errorCode, scanner_->filename(), orc_column_id_); : return scanner_- > nit: fits single line again Done -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 13:14:45 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15104 to look at the new patch set (#7). Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. IMPALA-9228: ORC scanner reads rows into scratch batch Because of performance considerations this change enhances ORC scanner to populate a scratch batch on a column-by-column manner using data from the column readers. Once this is done the parquet code was reused to apply runtime filter and conjuncts and to populate the outgoing row batch. This approach reduces the number of virtual function calls and takes advantage of the columnar orientation of the data to enhance scan performance. Additionally, introducing the scratch batch concept also opens the door for codegen runtime filtering and applying conjuncts. Tesing: - Re-run the full test suite to verify that no regression is introduced. - Checked the performance impact by running TPCH workload on a scale 25 database using single_node_perf_run.py. The total query runtime is decreased by 0-20% depending on how scan heavy the particular query was. The more scan heavy the query is the more performance gain I observe. Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/CMakeLists.txt R be/src/exec/hdfs-columnar-scanner-ir.cc A be/src/exec/hdfs-columnar-scanner.cc A be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h R be/src/exec/scratch-tuple-batch.h 15 files changed, 426 insertions(+), 144 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/15104/7 -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 7 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6360: Don't show full query statement on Impala webUI by default
Adam Tamas has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/15288 ) Change subject: IMPALA-6360: Don't show full query statement on Impala webUI by default .. IMPALA-6360: Don't show full query statement on Impala webUI by default Added the ‘query_stmt_size’ flag to impala-server.cc with default value of 250 and modified the ‘ImpalaHttpHandler::QueryStateToJson()’ to truncate the end of the statements if they are too long. Testing: -manual testing in the WebUI. -added 'test_query_stmt()' to test_web_pages.py Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 --- M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M tests/webserver/test_web_pages.py 3 files changed, 34 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/15288/6 -- To view, visit http://gerrit.cloudera.org:8080/15288 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib7109a0be5d1022b4f8d6e72441cf5dc1dc42605 Gerrit-Change-Number: 15288 Gerrit-PatchSet: 6 Gerrit-Owner: Adam Tamas Gerrit-Reviewer: Adam Tamas Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tamas Mate
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/15104 ) Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. Patch Set 6: PS6 is a rebase with master to resolve conflict with the ORC string allocations enhancement. -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Mon, 02 Mar 2020 12:57:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9228: ORC scanner reads rows into scratch batch
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/15104 to look at the new patch set (#6). Change subject: IMPALA-9228: ORC scanner reads rows into scratch batch .. IMPALA-9228: ORC scanner reads rows into scratch batch Because of performance considerations this change enhances ORC scanner to populate a scratch batch on a column-by-column manner using data from the column readers. Once this is done the parquet code was reused to apply runtime filter and conjuncts and to populate the outgoing row batch. This approach reduces the number of virtual function calls and takes advantage of the columnar orientation of the data to enhance scan performance. Additionally, introducing the scratch batch concept also opens the door for codegen runtime filtering and applying conjuncts. Tesing: - Re-run the full test suite to verify that no regression is introduced. - Checked the performance impact by running TPCH workload on a scale 25 database using single_node_perf_run.py. The total query runtime is decreased by 0-20% depending on how scan heavy the particular query was. The more scan heavy the query is the more performance gain I observe. Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca --- M be/src/codegen/gen_ir_descriptions.py M be/src/codegen/impala-ir.cc M be/src/exec/CMakeLists.txt R be/src/exec/hdfs-columnar-scanner-ir.cc A be/src/exec/hdfs-columnar-scanner.cc A be/src/exec/hdfs-columnar-scanner.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-orc-scanner.h M be/src/exec/hdfs-scanner.h M be/src/exec/orc-column-readers.cc M be/src/exec/orc-column-readers.h M be/src/exec/parquet/CMakeLists.txt M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-scanner.h R be/src/exec/scratch-tuple-batch.h 15 files changed, 423 insertions(+), 140 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/15104/6 -- To view, visit http://gerrit.cloudera.org:8080/15104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I56db0325dee283d73742ebbae412d19693fac0ca Gerrit-Change-Number: 15104 Gerrit-PatchSet: 6 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
