[Impala-ASF-CR] IMPALA-10675: Exclude hive-exec from ranger-plugins-audit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17339 ) Change subject: IMPALA-10675: Exclude hive-exec from ranger-plugins-audit .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8634/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c Gerrit-Change-Number: 17339 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Sat, 24 Apr 2021 01:35:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10675: Exclude hive-exec from ranger-plugins-audit
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17339 Change subject: IMPALA-10675: Exclude hive-exec from ranger-plugins-audit .. IMPALA-10675: Exclude hive-exec from ranger-plugins-audit A recent patch for RANGER-1837 results in the artifact of ranger-plugins-audit pulling in from the remote repository a hive-exec which may bring in unnecessary or even conflicting dependencies. The current ranger-plugins-audit that is pulled in has not had this change. To prevent the hive-exec from bringing in undesired dependencies when we bump up CDP_BUILD_NUMBER again, this patch excludes hive-exec when the dependency of ranger-plugins-audit is added. Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c --- M fe/pom.xml 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/17339/1 -- To view, visit http://gerrit.cloudera.org:8080/17339 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I459360028e12dabbdbaff2250f1b712b6f165f5c Gerrit-Change-Number: 17339 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 21: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 21 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 24 Apr 2021 00:03:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10650: Bailout min/max filters in hash join builder early
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17295 ) Change subject: IMPALA-10650: Bailout min/max filters in hash join builder early .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8633/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17295 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I193646e7acfdd3023f7c947d8107da58a1f41183 Gerrit-Change-Number: 17295 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Fri, 23 Apr 2021 23:14:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10650: Bailout min/max filters in hash join builder early
Qifan Chen has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/17295 ) Change subject: IMPALA-10650: Bailout min/max filters in hash join builder early .. IMPALA-10650: Bailout min/max filters in hash join builder early This change set addresses the weakness in population min/max filters in the hash join builder by periodically measuring the usefulness of each such filter and set the 'always_true_' flag to true. Once set to true, the insertion to such a filter will completely skip the steps from the evaluation of the value from a row to the verification of the value in the min/max range. This optimization is LLVM-codeded. In addition, a new flag 'is_min_max_value_present' is added to TRuntimeFilterTargetDesc to indicate whether the min/max column stats is present in the query plan. The flag eliminates the need to check the presence of min/max stats for every row in runtime. Testing: 1. Ran core test; 2. Ran performance test (TBD). Change-Id: I193646e7acfdd3023f7c947d8107da58a1f41183 --- M be/src/codegen/gen_ir_descriptions.py M be/src/exec/filter-context.cc M be/src/exec/filter-context.h M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/runtime/runtime-filter-ir.cc M be/src/util/min-max-filter-ir.cc M be/src/util/min-max-filter.cc M be/src/util/min-max-filter.h M common/thrift/PlanNodes.thrift M fe/src/main/java/org/apache/impala/planner/RuntimeFilterGenerator.java M fe/src/main/java/org/apache/impala/util/TColumnValueUtil.java 12 files changed, 368 insertions(+), 150 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/17295/13 -- To view, visit http://gerrit.cloudera.org:8080/17295 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I193646e7acfdd3023f7c947d8107da58a1f41183 Gerrit-Change-Number: 17295 Gerrit-PatchSet: 13 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-10644: RangerAuthorizationFactory cannot be instantiated
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17282 ) Change subject: IMPALA-10644: RangerAuthorizationFactory cannot be instantiated .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/17282/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/17282/1//COMMIT_MSG@15 PS1, Line 15: Testing: [WIP] > Please fill this in whenever the tests are done Unfortunately, I am not able to reproduce the issue in master branch. I am trying to reproduce it on a different environment and will update the details once they are done. http://gerrit.cloudera.org:8080/#/c/17282/1/fe/pom.xml File fe/pom.xml: http://gerrit.cloudera.org:8080/#/c/17282/1/fe/pom.xml@a114 PS1, Line 114: : : : > Nit: Can you add a tombstone comment saying that we are intentionally not e Done http://gerrit.cloudera.org:8080/#/c/17282/1/fe/pom.xml@a138 PS1, Line 138: : : : > Nit: Same deal here: please add a tombstone comment saying that we are inte Done -- To view, visit http://gerrit.cloudera.org:8080/17282 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1b6953b84fd28bb75f97516a3b7f40cd0a12af41 Gerrit-Change-Number: 17282 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Fri, 23 Apr 2021 22:54:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10656: Fire insert events before commit
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/17313 ) Change subject: IMPALA-10656: Fire insert events before commit .. Patch Set 12: Code-Review+1 (1 comment) http://gerrit.cloudera.org:8080/#/c/17313/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17313/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4876 PS12, Line 4876: These ACID_WRITE events are collected by HMS and become :* visible during commit I see that you added this. But I am not sure if this is correct. HMS metadata is not transactional and hence it has no awareness of when the events needs to be made visible. The events will be visible as soon as the HMS API succeeds AFAIK. It is upto the clients to determine if the ACID_WRITE event need to be processed or not based on the fact that the writeId in the event is committed or not. This commit or abort also generated a COMMIT_TXN or ABORT_TXN. But now may be we are going into implementation details. -- To view, visit http://gerrit.cloudera.org:8080/17313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9 Gerrit-Change-Number: 17313 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 22:30:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10611: Fix flakiness in test wide row
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17324 ) Change subject: IMPALA-10611: Fix flakiness in test_wide_row .. IMPALA-10611: Fix flakiness in test_wide_row test_wide_row has been intermittently failed with "Failed to allocate row batch" error message. This is due to recent change in IMPALA-9856 that add query option max_row_size=10MB without raising the mem_limit. This patch fix the flakiness by increasing the mem_limit from 100 MB to 132 MB to account for 32 MB reservation needed by BufferedPlanRootSink. Testing: - Loop the test in local dev machine. Change-Id: Ie1f0b7d4d6b3a875d9b408f057d46fdbdbdf2a34 Reviewed-on: http://gerrit.cloudera.org:8080/17324 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_scanners.py 1 file changed, 4 insertions(+), 2 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/17324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie1f0b7d4d6b3a875d9b408f057d46fdbdbdf2a34 Gerrit-Change-Number: 17324 Gerrit-PatchSet: 3 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10611: Fix flakiness in test wide row
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17324 ) Change subject: IMPALA-10611: Fix flakiness in test_wide_row .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/17324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie1f0b7d4d6b3a875d9b408f057d46fdbdbdf2a34 Gerrit-Change-Number: 17324 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 21:12:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 20: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8632/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 20 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 18:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 21: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7096/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 21 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 18:16:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 20: (5 comments) PS16 adds a few more unicode related fix (this time to printing errors from Impala) http://gerrit.cloudera.org:8080/#/c/17170/16/bin/impala-shell.sh File bin/impala-shell.sh: http://gerrit.cloudera.org:8080/#/c/17170/16/bin/impala-shell.sh@30 PS16, Line 30: > This dir no longer exists, which causes the data-loading failure that impal Thanks a lot for spotting this! http://gerrit.cloudera.org:8080/#/c/17170/16/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/17170/16/shell/impala_client.py@1149 PS16, Line 1149: ry_handle, False > nit: isinstance(row, str) Done http://gerrit.cloudera.org:8080/#/c/17170/16/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/17170/16/shell/impala_shell.py@1143 PS16, Line 1143: > nit: isinstance(col, str) Done http://gerrit.cloudera.org:8080/#/c/17170/16/shell/shell_output.py File shell/shell_output.py: http://gerrit.cloudera.org:8080/#/c/17170/16/shell/shell_output.py@44 PS16, Line 44: isinstance(entry, > nit: isinstance(entry, str) Done http://gerrit.cloudera.org:8080/#/c/17170/16/shell/shell_output.py@90 PS16, Line 90: isinstance(val, unic > nit: isinstance(val, unicode) Done -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 20 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 18:11:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 20: (1 comment) http://gerrit.cloudera.org:8080/#/c/17170/20/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/17170/20/shell/impala_client.py@85 PS20, Line 85: def utf8_decode_if_needed(val): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 20 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 18:10:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Hello Quanlong Huang, Tamas Mate, Qifan Chen, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17170 to look at the new patch set (#20). Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. IMPALA-7825: Upgrade Thrift version to 0.11.0 Before this patch Impala mainly used Thrift 0.9.3, but it was possible to compile Impala shell with Thrift 0.11.0, so the 0.11.0 Thrift lib was already included in the toolchain. Most of the changes are related to replacing boost:: with std:: shared_ptr-s in cpp code (this is a continuation of patch by Sahil). The Thrift upgrade also needs an Impyla release with Thrift 0.11.0, as Impala's test framework relies on Impyla. A thrift_sasl release is also needed, because it currently pins Thrift version to 0.9.3 for Python 2. The current patch uses alpha releases from Impyla and thrift_sasl that use thrift 0.11.0. Notable side effects: - old logic to compile thrift for impala-shell with 0.11.0 was removed - impala_shell's utf8 handling had to be updated as the new 0.11.0 compilation happens with no_utf8strings. This also made things a bit faster, e.g the following is ~0.22s instead of ~0.25 shell/impala_shell.py \ -B -q "select * from functional_parquet.alltypes;" > /dev/null - THRIFT-3921 changed the stream operators to print an enum's name instead of its number, leading to slightly different messages in some cases. - "templates" was added to the thift generator's parameters to avoid a compilation issue (related to IMPALA-10600). I didn't notice any change in compilation time. This option generated .tcc files with templetized readers/writers for Thrift types. Currently we don't use these, but they could potentially speed up (de)serialization. Testing: - ran Impyla's test suite with Python 2 and 3 - ran core tests Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 --- M CMakeLists.txt M be/src/benchmarks/network-perf-benchmark.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalogd-main.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc M be/src/rpc/hs2-http-test.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M be/src/transport/TSaslClientTransport.cpp M be/src/transport/TSaslClientTransport.h M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslServerTransport.h M be/src/transport/TSaslTransport.cpp M be/src/transport/TSaslTransport.h M be/src/util/parquet-reader.cc M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/impala-shell.sh M bin/set-pythonpath.sh M common/thrift/CMakeLists.txt M infra/python/deps/requirements.txt M java/pom.xml M shell/ext-py/thrift_sasl-0.4.2/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh M shell/shell_output.py M tests/beeswax/impala_beeswax.py M tests/conftest.py M tests/query_test/test_observability.py M tests/shell/util.py 57 files changed, 234 insertions(+), 303 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/17170/20 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 20 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10669: Limit memory during dataload in Docker-based tests
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17331 ) Change subject: IMPALA-10669: Limit memory during dataload in Docker-based tests .. Patch Set 1: I believe applications can find out how much memory is available to their container by looking at cgroup structures. Impala does that, so maybe Hive/Tez do the same. If they do, they would know they are running in a container and use the smaller memory limit. If they don't, then this won't work. -- To view, visit http://gerrit.cloudera.org:8080/17331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5100589ffc54184b6ca13db139cb983175c934eb Gerrit-Change-Number: 17331 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Fri, 23 Apr 2021 16:16:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10669: Limit memory during dataload in Docker-based tests
Laszlo Gaal has abandoned this change. ( http://gerrit.cloudera.org:8080/17331 ) Change subject: IMPALA-10669: Limit memory during dataload in Docker-based tests .. Abandoned Additional tests proved that this is not sufficient on its own. Memory cgroup documentation also suggests that limiting container memory will not be reflected in the APIs userland programs routinely call for querying system/free memory (e.g. free(1)), so this is unlikely to influence how Hive & Tez manage Tez container memory. -- To view, visit http://gerrit.cloudera.org:8080/17331 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: abandon Gerrit-Change-Id: I5100589ffc54184b6ca13db139cb983175c934eb Gerrit-Change-Number: 17331 Gerrit-PatchSet: 1 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 19: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7094/ -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 19 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 15:38:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10611: Fix flakiness in test wide row
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17324 ) Change subject: IMPALA-10611: Fix flakiness in test_wide_row .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/17324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie1f0b7d4d6b3a875d9b408f057d46fdbdbdf2a34 Gerrit-Change-Number: 17324 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 15:26:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10611: Fix flakiness in test wide row
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17324 ) Change subject: IMPALA-10611: Fix flakiness in test_wide_row .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7095/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/17324 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie1f0b7d4d6b3a875d9b408f057d46fdbdbdf2a34 Gerrit-Change-Number: 17324 Gerrit-PatchSet: 2 Gerrit-Owner: Riza Suminto Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Riza Suminto Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 15:26:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10656: Fire insert events before commit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17313 ) Change subject: IMPALA-10656: Fire insert events before commit .. Patch Set 12: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8631/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9 Gerrit-Change-Number: 17313 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 15:21:01 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17262 ) Change subject: WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8630/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie865efd4f0c11b9e111fb94f77d084bf6ee20792 Gerrit-Change-Number: 17262 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 23 Apr 2021 15:03:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10656: Fire insert events before commit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17313 ) Change subject: IMPALA-10656: Fire insert events before commit .. Patch Set 12: (1 comment) http://gerrit.cloudera.org:8080/#/c/17313/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/17313/12/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4881 PS12, Line 4881:* https://github.com/apache/hive/blob/25892ea409/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3251 line too long (114 > 90) -- To view, visit http://gerrit.cloudera.org:8080/17313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9 Gerrit-Change-Number: 17313 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 15:01:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10656: Fire insert events before commit
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/17313 )
Change subject: IMPALA-10656: Fire insert events before commit
..
Patch Set 12:
(3 comments)
Thanks Vihang for the detailed comments in the comments :) I only made small
changes to your suggestions.
http://gerrit.cloudera.org:8080/#/c/17313/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17313/8/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4902
PS8, Line 4902: .format("Invalid transaction id %s for generating
insert events on table %s",
: txnId, table.getFullName()));
: Preconditions.checkState(!isTransactional || writeId > 0,
: String.format("Invalid write id %s for generating
insert events on table %s",
: writeId, table.getFullName()));
:
> Okay, It took me a bit of poking around in Hive code to understand if we ar
Done
http://gerrit.cloudera.org:8080/#/c/17313/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java:
http://gerrit.cloudera.org:8080/#/c/17313/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4867
PS11, Line 4867: TODO: I am not sure that it is the right thing to connect
event polling and
:*
> Can you add a comment about the behavior of the transactional tables here.
Done
http://gerrit.cloudera.org:8080/#/c/17313/11/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@4885
PS11, Line 4885: aram addedPartitio
> Can you please add the following to the comment -- "This would show up in H
Done
--
To view, visit http://gerrit.cloudera.org:8080/17313
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9
Gerrit-Change-Number: 17313
Gerrit-PatchSet: 12
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vihang Karajgaonkar
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 23 Apr 2021 15:02:46 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10656: Fire insert events before commit
Hello Vihang Karajgaonkar, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17313 to look at the new patch set (#12). Change subject: IMPALA-10656: Fire insert events before commit .. IMPALA-10656: Fire insert events before commit Before this fix Impala committed an insert first, then reloaded the table from HMS, and generated the insert events based on the difference between the two snapshots. (e.g. which file was not present in the old snapshot but are there in the new one). Hive replication expects the insert events before the commit, so this may potentially lead to issues there. The solution is to collect the new files during the insert in the backend, and send the insert events based on this file set. This wasn't very hard to do as we were already collecting the files in some cases: - to move them from staging dir to their final location in case of non-partitioned tables - to write the file list to snapshot files in case of Iceberg tables This patch unifies the paths above and collects all information about the created files regardless of the table type. Testing: - no new tests, insert events were already covered in test_event_processing.py and MetastoreEventsProcessorTest.java - ran core tests Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9 --- M be/src/exec/hbase-table-sink.cc M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-text-table-writer.cc M be/src/exec/output-partition.h M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/runtime/dml-exec-state.cc M be/src/runtime/dml-exec-state.h M be/src/service/client-request-state.cc M common/protobuf/control_service.proto M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 12 files changed, 248 insertions(+), 226 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/17313/12 -- To view, visit http://gerrit.cloudera.org:8080/17313 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2ed812dbcb5f55efff3a910a3daeeb76cd3295b9 Gerrit-Change-Number: 17313 Gerrit-PatchSet: 12 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17262 ) Change subject: WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types .. Patch Set 6: (3 comments) http://gerrit.cloudera.org:8080/#/c/17262/6/tests/query_test/test_parquet_bloom_filter.py File tests/query_test/test_parquet_bloom_filter.py: http://gerrit.cloudera.org:8080/#/c/17262/6/tests/query_test/test_parquet_bloom_filter.py@28 PS6, Line 28: g flake8: E126 continuation line over-indented for hanging indent http://gerrit.cloudera.org:8080/#/c/17262/6/tests/query_test/test_parquet_bloom_filter.py@126 PS6, Line 126: * flake8: E226 missing whitespace around arithmetic operator http://gerrit.cloudera.org:8080/#/c/17262/6/tests/query_test/test_parquet_bloom_filter.py@145 PS6, Line 145: b flake8: F841 local variable 'bloom_filter_header' is assigned to but never used -- To view, visit http://gerrit.cloudera.org:8080/17262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie865efd4f0c11b9e111fb94f77d084bf6ee20792 Gerrit-Change-Number: 17262 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 23 Apr 2021 14:46:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10672: Fix non-deterministic RANK queries in tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17333 ) Change subject: IMPALA-10672: Fix non-deterministic RANK queries in tests .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8629/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17333 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I396549f2ba3928cb4db942def9fae6504266d344 Gerrit-Change-Number: 17333 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Kurt Deschler Gerrit-Comment-Date: Fri, 23 Apr 2021 14:46:27 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types
Daniel Becker has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/17262 ) Change subject: WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types .. WIP - IMPALA-10642: Write support for Parquet Bloom filters - most common types This change adds support for writing Parquet Bloom filters for the types for which read support was added in IMPALA-10640. Writing of Parquet Bloom filters can be controlled by the 'parquet_bloom_filter_write' query option which has the following possible values: NEVER - never write Parquet Bloom filters TBL_PROPS - write Parquet Bloom filters as set in table properties IF_NO_DICT - write Parquet Bloom filters if the row group is not fully dictionary encoded ALWAYS - always write Parquet Bloom filters, even if the row group is fully dictionary encoded TODO: Implement table properties involving Parquet Bloom filters. TODO: Decide size of Parquet Bloom filter based on NDV heuristics or configuration. Testing: - Added a test in tests/query_test/test_parquet_bloom_filter.py that uses Impala to write the same table as in the test file 'testdata/data/parquet-bloom-filtering.parquet' and checks whether the Parquet Bloom filter header and bitset are identical. Change-Id: Ie865efd4f0c11b9e111fb94f77d084bf6ee20792 --- M be/src/exec/parquet/hdfs-parquet-scanner.cc M be/src/exec/parquet/hdfs-parquet-table-writer.cc M be/src/exec/parquet/hdfs-parquet-table-writer.h M be/src/exec/parquet/parquet-bloom-filter-util.cc M be/src/exec/parquet/parquet-bloom-filter-util.h M be/src/service/query-options.cc M be/src/service/query-options.h M be/src/util/debug-util.cc M be/src/util/debug-util.h M be/src/util/dict-encoding.h M be/src/util/parquet-bloom-filter-test.cc M be/src/util/parquet-bloom-filter.cc M be/src/util/parquet-bloom-filter.h M common/thrift/ImpalaService.thrift M common/thrift/Query.thrift M tests/query_test/test_parquet_bloom_filter.py 16 files changed, 486 insertions(+), 30 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/62/17262/6 -- To view, visit http://gerrit.cloudera.org:8080/17262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie865efd4f0c11b9e111fb94f77d084bf6ee20792 Gerrit-Change-Number: 17262 Gerrit-PatchSet: 6 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10672: Fix non-deterministic RANK queries in tests
Kurt Deschler has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17333 Change subject: IMPALA-10672: Fix non-deterministic RANK queries in tests .. IMPALA-10672: Fix non-deterministic RANK queries in tests This patch adds additional columns to RANK queries that were producing varying results so that the RANK ordering is now strong. Results have been updated to reflect the new ordering. Testing: Ran test_limit_pushdown_analytic many times in a loop to verify stable results. Change-Id: I396549f2ba3928cb4db942def9fae6504266d344 --- M testdata/workloads/functional-query/queries/limit-pushdown-analytic.test M testdata/workloads/tpch/queries/limit-pushdown-analytic.test 2 files changed, 40 insertions(+), 40 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/17333/1 -- To view, visit http://gerrit.cloudera.org:8080/17333 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I396549f2ba3928cb4db942def9fae6504266d344 Gerrit-Change-Number: 17333 Gerrit-PatchSet: 1 Gerrit-Owner: Kurt Deschler
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 19: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7094/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 19 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 09:57:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8628/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 19 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 09:28:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 18: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8627/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 09:23:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 17: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/8626/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 17 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 09:22:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Hello Quanlong Huang, Tamas Mate, Qifan Chen, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17170 to look at the new patch set (#19). Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. IMPALA-7825: Upgrade Thrift version to 0.11.0 Before this patch Impala mainly used Thrift 0.9.3, but it was possible to compile Impala shell with Thrift 0.11.0, so the 0.11.0 Thrift lib was already included in the toolchain. Most of the changes are related to replacing boost:: with std:: shared_ptr-s in cpp code (this is a continuation of patch by Sahil). The Thrift upgrade also needs an Impyla release with Thrift 0.11.0, as Impala's test framework relies on Impyla. A thrift_sasl release is also needed, because it currently pins Thrift version to 0.9.3 for Python 2. The current patch uses alpha releases from Impyla and thrift_sasl that use thrift 0.11.0. Notable side effects: - old logic to compile thrift for impala-shell with 0.11.0 was removed - impala_shell's utf8 handling had to be updated as the new 0.11.0 compilation happens with no_utf8strings. This also made things a bit faster, e.g the following is ~0.22s instead of ~0.25 shell/impala_shell.py \ -B -q "select * from functional_parquet.alltypes;" > /dev/null - THRIFT-3921 changed the stream operators to print an enum's name instead of its number, leading to slightly different messages in some cases. - "templates" was added to the thift generator's parameters to avoid a compilation issue (related to IMPALA-10600). I didn't notice any change in compilation time. This option generated .tcc files with templetized readers/writers for Thrift types. Currently we don't use these, but they could potentially speed up (de)serialization. Testing: - ran Impyla's test suite with Python 2 and 3 - ran core tests Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 --- M CMakeLists.txt M be/src/benchmarks/network-perf-benchmark.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalogd-main.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc M be/src/rpc/hs2-http-test.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M be/src/transport/TSaslClientTransport.cpp M be/src/transport/TSaslClientTransport.h M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslServerTransport.h M be/src/transport/TSaslTransport.cpp M be/src/transport/TSaslTransport.h M be/src/util/parquet-reader.cc M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/impala-shell.sh M bin/set-pythonpath.sh M common/thrift/CMakeLists.txt M infra/python/deps/requirements.txt M java/pom.xml M shell/ext-py/thrift_sasl-0.4.2/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh M shell/shell_output.py M tests/beeswax/impala_beeswax.py M tests/conftest.py M tests/query_test/test_observability.py M tests/shell/util.py 57 files changed, 225 insertions(+), 301 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/17170/19 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 19 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17170 ) Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. Patch Set 18: (2 comments) http://gerrit.cloudera.org:8080/#/c/17170/18/shell/shell_output.py File shell/shell_output.py: http://gerrit.cloudera.org:8080/#/c/17170/18/shell/shell_output.py@44 PS18, Line 44: flake8: E203 whitespace before ',' http://gerrit.cloudera.org:8080/#/c/17170/18/shell/shell_output.py@44 PS18, Line 44: , flake8: E231 missing whitespace after ',' -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 23 Apr 2021 09:04:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Hello Quanlong Huang, Tamas Mate, Qifan Chen, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17170 to look at the new patch set (#18). Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. IMPALA-7825: Upgrade Thrift version to 0.11.0 Before this patch Impala mainly used Thrift 0.9.3, but it was possible to compile Impala shell with Thrift 0.11.0, so the 0.11.0 Thrift lib was already included in the toolchain. Most of the changes are related to replacing boost:: with std:: shared_ptr-s in cpp code (this is a continuation of patch by Sahil). The Thrift upgrade also needs an Impyla release with Thrift 0.11.0, as Impala's test framework relies on Impyla. A thrift_sasl release is also needed, because it currently pins Thrift version to 0.9.3 for Python 2. The current patch uses alpha releases from Impyla and thrift_sasl that use thrift 0.11.0. Notable side effects: - old logic to compile thrift for impala-shell with 0.11.0 was removed - impala_shell's utf8 handling had to be updated as the new 0.11.0 compilation happens with no_utf8strings. This also made things a bit faster, e.g the following is ~0.22s instead of ~0.25 shell/impala_shell.py \ -B -q "select * from functional_parquet.alltypes;" > /dev/null - THRIFT-3921 changed the stream operators to print an enum's name instead of its number, leading to slightly different messages in some cases. - "templates" was added to the thift generator's parameters to avoid a compilation issue (related to IMPALA-10600). I didn't notice any change in compilation time. This option generated .tcc files with templetized readers/writers for Thrift types. Currently we don't use these, but they could potentially speed up (de)serialization. Testing: - ran Impyla's test suite with Python 2 and 3 - ran core tests Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 --- M CMakeLists.txt M be/src/benchmarks/network-perf-benchmark.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalogd-main.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc M be/src/rpc/hs2-http-test.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M be/src/transport/TSaslClientTransport.cpp M be/src/transport/TSaslClientTransport.h M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslServerTransport.h M be/src/transport/TSaslTransport.cpp M be/src/transport/TSaslTransport.h M be/src/util/parquet-reader.cc M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/impala-shell.sh M bin/set-pythonpath.sh M common/thrift/CMakeLists.txt M infra/python/deps/requirements.txt M java/pom.xml M shell/ext-py/thrift_sasl-0.4.2/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh M shell/shell_output.py M tests/beeswax/impala_beeswax.py M tests/conftest.py M tests/query_test/test_observability.py M tests/shell/util.py 57 files changed, 225 insertions(+), 301 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/17170/18 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 18 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7825: Upgrade Thrift version to 0.11.0
Hello Quanlong Huang, Tamas Mate, Qifan Chen, Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/17170 to look at the new patch set (#17). Change subject: IMPALA-7825: Upgrade Thrift version to 0.11.0 .. IMPALA-7825: Upgrade Thrift version to 0.11.0 Before this patch Impala mainly used Thrift 0.9.3, but it was possible to compile Impala shell with Thrift 0.11.0, so the 0.11.0 Thrift lib was already included in the toolchain. Most of the changes are related to replacing boost:: with std:: shared_ptr-s in cpp code (this is a continuation of patch by Sahil). The Thrift upgrade also needs an Impyla release with Thrift 0.11.0, as Impala's test framework relies on Impyla. A thrift_sasl release is also needed, because it currently pins Thrift version to 0.9.3 for Python 2. The current patch uses alpha releases from Impyla and thrift_sasl that use thrift 0.11.0. Notable side effects: - old logic to compile thrift for impala-shell with 0.11.0 was removed - impala_shell's utf8 handling had to be updated as the new 0.11.0 compilation happens with no_utf8strings. This also made things a bit faster, e.g the following is ~0.22s instead of ~0.25 shell/impala_shell.py \ -B -q "select * from functional_parquet.alltypes;" > /dev/null - THRIFT-3921 changed the stream operators to print an enum's name instead of its number, leading to slightly different messages in some cases. - "templates" was added to the thift generator's parameters to avoid a compilation issue (related to IMPALA-10600). I didn't notice any change in compilation time. This option generated .tcc files with templetized readers/writers for Thrift types. Currently we don't use these, but they could potentially speed up (de)serialization. Testing: - ran Impyla's test suite with Python 2 and 3 - ran core tests Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 --- M CMakeLists.txt M be/src/benchmarks/network-perf-benchmark.cc M be/src/catalog/catalog-server.h M be/src/catalog/catalog-service-client-wrapper.h M be/src/catalog/catalog-util.cc M be/src/catalog/catalogd-main.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/auth-provider.h M be/src/rpc/authentication.cc M be/src/rpc/hs2-http-test.cc M be/src/rpc/thrift-client.h M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-thread.cc M be/src/rpc/thrift-thread.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/statestore/statestore-service-client-wrapper.h M be/src/statestore/statestore-subscriber-client-wrapper.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore-subscriber.h M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M be/src/testutil/in-process-servers.h M be/src/transport/THttpServer.cpp M be/src/transport/THttpServer.h M be/src/transport/THttpTransport.cpp M be/src/transport/THttpTransport.h M be/src/transport/TSaslClientTransport.cpp M be/src/transport/TSaslClientTransport.h M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslServerTransport.h M be/src/transport/TSaslTransport.cpp M be/src/transport/TSaslTransport.h M be/src/util/parquet-reader.cc M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/impala-shell.sh M bin/set-pythonpath.sh M common/thrift/CMakeLists.txt M infra/python/deps/requirements.txt M java/pom.xml M shell/ext-py/thrift_sasl-0.4.2/setup.py M shell/impala_client.py M shell/impala_shell.py M shell/make_shell_tarball.sh M shell/packaging/make_python_package.sh M shell/shell_output.py M tests/beeswax/impala_beeswax.py M tests/conftest.py M tests/query_test/test_observability.py M tests/shell/util.py 57 files changed, 225 insertions(+), 301 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/70/17170/17 -- To view, visit http://gerrit.cloudera.org:8080/17170 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idd13f177b4f7acc07872ea6399035aa180ef6ab6 Gerrit-Change-Number: 17170 Gerrit-PatchSet: 17 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tamas Mate Gerrit-Reviewer: Zoltan Borok-Nagy
