[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:35:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6494/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 4 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 4 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7329/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 2 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed when running in docker containers with a lower mem limit than host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra resource. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left into threasholds below. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account. - Modified the logic in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py' Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 9 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/3 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed in containers whose mem limit are lower than the host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra memories. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left memory into threasholds. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account. - Modified the logic in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py' Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 2 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test File testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test: http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test@278 PS3, Line 278: select * from : functional.alltypestiny a : where a.id in (select id from functional.alltypestiny limit 5) limit 5 > Normally, the error becomes larger with more number of tables joined as wel Not sure I follow. Can you file a follow up JIRA since this has already been merged? -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 22:40:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16291 ) Change subject: IMPALA-9180 (part 1): Remove legacy ImpalaInternalService .. IMPALA-9180 (part 1): Remove legacy ImpalaInternalService The legacy Thrift based Impala internal service has been deprecated and can be removed now. This patch removes ImpalaInternalService. All infrastructures around it are cleaned up, except one place for flag be_port. StatestoreSubscriber::subscriber_id consists be_port, but we cannot change format of subscriber_id now. This remaining be_port issue will be fixed in a succeeding patch (part 4). TQueryCtx.coord_address is changed to TQueryCtx.coord_hostname since the port in TQueryCtx.coord_address is set as be_port and is unused now. Also Rename TQueryCtx.coord_krpc_address as TQueryCtx.coord_ip_address. Testing: - Passed the exhaustive test. - Passed Quasar-L0 test. Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6 Reviewed-on: http://gerrit.cloudera.org:8080/16291 Reviewed-by: Thomas Tauber-Marshall Tested-by: Impala Public Jenkins --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/expr-benchmark.cc M be/src/exprs/expr-test.cc M be/src/exprs/utility-functions-ir.cc M be/src/rpc/impala-service-pool.cc M be/src/rpc/impala-service-pool.h M be/src/rpc/rpc-mgr.cc M be/src/rpc/rpc-mgr.h M be/src/rpc/thrift-server-test.cc D be/src/runtime/backend-client.h M be/src/runtime/client-cache-types.h M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/data-stream-test.cc M be/src/runtime/exec-env.cc M be/src/runtime/exec-env.h M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/initial-reservations.cc M be/src/runtime/query-exec-mgr.cc M be/src/runtime/query-state.cc M be/src/runtime/runtime-filter-bank.cc M be/src/runtime/test-env.cc M be/src/scheduling/scheduler-test-util.h M be/src/service/CMakeLists.txt M be/src/service/client-request-state.cc M be/src/service/control-service.cc D be/src/service/impala-internal-service.cc D be/src/service/impala-internal-service.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/impalad-main.cc M be/src/service/session-expiry-test.cc M be/src/testutil/in-process-servers.cc M be/src/testutil/in-process-servers.h M be/src/util/debug-util.cc M bin/generate_minidump_collection_testdata.py M common/thrift/ImpalaInternalService.thrift M tests/custom_cluster/test_blacklist.py M tests/custom_cluster/test_process_failures.py M tests/custom_cluster/test_query_retries.py M tests/custom_cluster/test_restart_services.py M tests/webserver/test_web_pages.py 42 files changed, 130 insertions(+), 351 deletions(-) Approvals: Thomas Tauber-Marshall: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/16291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6 Gerrit-Change-Number: 16291 Gerrit-PatchSet: 15 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16291 ) Change subject: IMPALA-9180 (part 1): Remove legacy ImpalaInternalService .. Patch Set 14: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6 Gerrit-Change-Number: 16291 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 22:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. IMPALA-3335: Allow single-node optimization with joins Enable the single-node optimization for queries with joins. Testing: * Ran exhaustive tests * Looped TPC-DS overnight with NUM_NODES=1 against an impalad mini-cluster with a single dedicated coordinator Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Reviewed-on: http://gerrit.cloudera.org:8080/16521 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test 3 files changed, 21 insertions(+), 11 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 4 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 22:36:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7328/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 22:00:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9867: Add Support for Spilling to S3: Milestone 1
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16318 )
Change subject: IMPALA-9867: Add Support for Spilling to S3: Milestone 1
..
Patch Set 13:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/hdfs-fs-cache.cc
File be/src/runtime/hdfs-fs-cache.cc:
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/hdfs-fs-cache.cc@103
PS13, Line 103: if (options != nullptr && !options->empty()) {
: for (auto option : *options) {
: hdfsBuilderConfSetStr(
: hdfs_builder, option.first.c_str(),
option.second.c_str());
: }
: }
does this actually work? were you able to confirm that non-default configs set
this way are actually picked up by the returned hdfsFS object?
according to the comment above, hdfsBuilderSetForceNewInstance has to be called
in order for hdfsBuilderConfSetStr to have an affect.
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/disk-io-mgr.h
File be/src/runtime/io/disk-io-mgr.h:
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/disk-io-mgr.h@370
PS13, Line 370: REMOTE_DFS_DISK_FILE_OPER_OFFSET,
: REMOTE_S3_DISK_FILE_OPER_OFFSET,
you cover this a bit in the commit message, but is the reason they were added
to de-couple HDFS/S3 scans / writes from HDFS/S3 tmp-file-mgr operations? I
guess that benefit is isolate between operations that spill vs. operations that
read / write data on behalf of queries?
it would be good to add some documentation here explaining why these two queues
are necessary and when to, for example, use REMOTE_S3_DISK_FILE_OPER_OFFSET vs.
REMOTE_S3_DISK_OFFSET
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/disk-io-mgr.cc
File be/src/runtime/io/disk-io-mgr.cc:
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/disk-io-mgr.cc@30
PS13, Line 30: #include "runtime/tmp-file-mgr-internal.h"
Why are we including the tmp-file-mgr here? I see a lot of references to the
tmp-file-mgr, but ideally I would think the disk-io-mgr and tmp-file-mgr would
remain separate systems. I see a lot of code referring to tmp_files in this
class as well, which I concerning.
I would think that the disk-io-mgr should just provide an interface to the
tmp-file-mgr.
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/disk-io-mgr.cc@241
PS13, Line 241: goto end;
I think we generally try to avoid 'goto' statements because they make the
control flow harder to follow. in this case the 'goto end' is only used once so
I don't think it is necessary.
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/request-ranges.h
File be/src/runtime/io/request-ranges.h:
http://gerrit.cloudera.org:8080/#/c/16318/13/be/src/runtime/io/request-ranges.h@113
PS13, Line 113: READ,
: WRITE,
: FETCH,
: UPLOAD,
not sure I understand the difference between all of these. fetch seems like it
would be the same thing as read, and upload seems like the same thing as write.
--
To view, visit http://gerrit.cloudera.org:8080/16318
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89
Gerrit-Change-Number: 16318
Gerrit-PatchSet: 13
Gerrit-Owner: Yida Wu
Gerrit-Reviewer: Abhishek Rawat
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Yida Wu
Gerrit-Comment-Date: Wed, 30 Sep 2020 21:55:55 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Wenzhe Zhou has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist In current ExecuterBlacklist class, we've keyed maps on a TNetworkAddress of a backend. To simply the logic for the class, changes it to key off of the UniqueIdPB backend-id, eg. refactor 'executor_list_' to no longer be a map list> and instead makes it a map. Also fixes a minor bug with the calculation of elapsed time when a backend that was on probation is re-blacklisted. Testing: - Passed test_blacklist.py and test_query_retries.py. - Passed exhaustive tests. Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 --- M be/src/scheduling/executor-blacklist.cc M be/src/scheduling/executor-blacklist.h 2 files changed, 68 insertions(+), 103 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/16506/6 -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 6 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h@136 PS5, Line 136: executor > nit. executor->'executor entry' done http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h@137 PS5, Line 137: blacklisted : /// and ones that are on probation > nit: either blacklisted or on probation. done http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc File be/src/scheduling/executor-blacklist.cc: http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc@51 PS5, Line 51: int64_t elapsed = MonotonicMillis() - entry.blacklist_time_ms; : entry.state = State::BLACKLISTED; : entry.blacklist_time_ms = MonotonicMillis(); : entry.cause = cause; > Should MonotonicMillis() be called only once? That is, in a single time poi done http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc@164 PS5, Line 164: "on probation") > Should we also check on NOT_BLACKLISTED here? No, don't need. -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 21:36:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16412 ) Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.cc File be/src/scheduling/remote-admission-control-client.cc: http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.cc@86 PS5, Line 86: KUDU_RETURN_IF_ERROR( : proxy->AdmitQuery(req, &resp, &rpc_controller), "AdmitQuery rpc failed"); : Status admit_status(resp.status()); : RETURN_IF_ERROR(admit_status); > is it necessary to hold the lock while making the RPC to Admit the query? t nvm, seems there is a separate client per query. -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 30 Sep 2020 21:14:00 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Sep 2020 21:01:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. IMPALA-10143: TestAcid.test_full_acid_original_files is flaky One of the test queries depended on the values of the row__id.rowid field. In rare cases the files are written differently by Hive which leads to different 'rowid' values for the rows. I took out the test for the 'rowid' values in that particular query. We have other tests for the 'rowid' field on static data files (from 'testdata/data'), therefore we still have coverage for that and they shouldn't be flaky. Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Reviewed-on: http://gerrit.cloudera.org:8080/16523 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/full-acid-original-file.test 1 file changed, 7 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 21: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6490/ -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 21 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 19:42:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10192: Filter out redundant AuthzAuditEvent's for column masking
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16524 ) Change subject: IMPALA-10192: Filter out redundant AuthzAuditEvent's for column masking .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7327/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1dbf65874003523b5176680e42f26fa2114c229b Gerrit-Change-Number: 16524 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 30 Sep 2020 19:41:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16412 ) Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc service .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h File be/src/scheduling/admission-control-service.h: http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h@78 PS5, Line 78: UniqueIdPB query_id; : UniqueIdPB coord_id; > can these both be const ref? nvm, I guess they need to be copied since they come from the AdmitQueryRequestPB -- To view, visit http://gerrit.cloudera.org:8080/16412 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I594fc593a27b24b6952e381a9bc1a9a5c6b757ae Gerrit-Change-Number: 16412 Gerrit-PatchSet: 5 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Comment-Date: Wed, 30 Sep 2020 19:34:36 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9930 (part 2): Introduce new admission control rpc service
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16412 )
Change subject: IMPALA-9930 (part 2): Introduce new admission control rpc
service
..
Patch Set 5:
(22 comments)
still going through it all, but my comments so far.
http://gerrit.cloudera.org:8080/#/c/16412/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/16412/5//COMMIT_MSG@12
PS5, Line 12: This patch adds some simple configuration flags that make it
possible
not sure I understand the plan here. this patch adds flags is
--is_admission_controller to exec-env, which if set will expose the
AdmissionControllerService. what about this will change exactly in IMPALA-9975?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h
File be/src/scheduling/admission-control-service.h:
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h@76
PS5, Line 76: AdmitQueryRequstPB
nit: typo
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h@78
PS5, Line 78: UniqueIdPB query_id;
: UniqueIdPB coord_id;
can these both be const ref?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h@105
PS5, Line 105: RuntimeProfile* summary_profile;
might be worth mentioning that this is passed to
AdmissionController::SubmitForAdmission, which is responsible for updating it.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.h@118
PS5, Line 118: fo
nit: typo
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc
File be/src/scheduling/admission-control-service.cc:
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@85
PS5, Line 85: Adqmission
nit: typo
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@142
PS5, Line 142: VLOG(1) << "GetQueryStatus " << req->query_id();
might be too verbose to log.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@155
PS5, Line 155: 100
why wait at all? won't waiting tie up one of the RPC threads? I think the
client already waits 100 ms.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@159
PS5, Line 159: DCHECK(query_info->admit_status.ok());
is this expected? do want to DCHECK if the admit_status is not Status::OK()? if
AC rejects the query due to cancellation, won't it returned a Status::CANCELLED?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@197
PS5, Line 197: lock_guard l(query_info->lock);
why does the lock need to be acquired here?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@209
PS5, Line 209: VLOG(1) << "ReleaseQueryBackends: query_id=" <<
req->query_id();
might be too verbose to log
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@219
PS5, Line 219: lock_guard l(query_info->lock);
why does the lock need to be acquired here?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-control-service.cc@249
PS5, Line 249: {query_info->query_id, query_info->coord_id,
query_info->query_exec_request,
: query_info->query_options,
query_info->summary_profile,
: query_info->blacklisted_executor_addresses},
might make the code easier to read if this just created a AdmissionRequest and
then passed it in.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-controller.h
File be/src/scheduling/admission-controller.h:
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-controller.h@316
PS5, Line 316: (usually they
: /// are owned by the ClientRequestState).
might need to revise this now.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/admission-controller.h@345
PS5, Line 345: Status WaitOnQueued(const UniqueIdPB& query_id,
not clear to me what the returned Status is suppose to indicate.
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.h
File be/src/scheduling/remote-admission-control-client.h:
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.h@53
PS5, Line 53: std::mutex lock_;
what is the lock used to protect?
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.cc
File be/src/scheduling/remote-admission-control-client.cc:
http://gerrit.cloudera.org:8080/#/c/16412/5/be/src/scheduling/remote-admission-control-client.cc@63
PS5, Line 63: KrpcSerializer serializer;
: int sidecar_idx1;
: RETURN_IF_ERROR(
: serializer
[Impala-ASF-CR] IMPALA-10192: Filter out redundant AuthzAuditEvent's for column masking
Fang-Yu Rao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16524 Change subject: IMPALA-10192: Filter out redundant AuthzAuditEvent's for column masking .. IMPALA-10192: Filter out redundant AuthzAuditEvent's for column masking We found that Ranger would generate an AuthzAuditEvent as long as there exists a column masking policy corresponding to the column even though the policy does not apply to the requesting user. This resulted in an IllegalStateException if a user "A" submits a SELECT query against a table that has a column specified in a column masking policy when the policy does not apply to "A", i.e., the field of 'Select User' for this policy in the Ranger web UI does not contain "A". For such an AuthzAuditEvent, its field of 'accessType' will not be one of the supported mask types since its corresponding accessResult.isMaskEnabled() would evaluates to false, indicating that there is no matching column masking policy associated with the user "A" and thus the AuthzAuditEvent will not be post-processed by Impala in RangerAuthorizationCheker#createColumnMask(). But since we did not filter out such an AuthzAuditEvent when it was generated and returned from RangerBasePlugin#evalDataMaskPolicies(), we failed the check that requires every AuthzAuditEvent be column masking-related in RangerAuthorizationContext#stashAuditEvents(). To address this issue, in this patch we filter out such an AuthzAuditEvent after each call to RangerBasePlugin#evalDataMaskPolicies() so that no redundant AuthzAuditEvent is generated. Furthermore, we also revise all the checks for the generated AuthzAuditEvent's due to the evaluation of column masking policies so that a failed check would not fail the query but only result in an entry in the log file. Testing: - Added a new column masking policy associated with a non-matching user in RangerAuditLogTest#testAuditsForColumnMasking() to verify that the redundant AuthzAuditEvent is removed. - Verified that the patch passes the exhaustive tests in the DEBUG build. Change-Id: I1dbf65874003523b5176680e42f26fa2114c229b --- M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationChecker.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationContext.java M fe/src/test/java/org/apache/impala/authorization/ranger/RangerAuditLogTest.java 3 files changed, 53 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/24/16524/1 -- To view, visit http://gerrit.cloudera.org:8080/16524 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I1dbf65874003523b5176680e42f26fa2114c229b Gerrit-Change-Number: 16524 Gerrit-PatchSet: 1 Gerrit-Owner: Fang-Yu Rao Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fang-Yu Rao Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 5: (4 comments) Sorry I was not able to finish the review before Thomas' +2. http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h@136 PS5, Line 136: executor nit. executor->'executor entry' http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.h@137 PS5, Line 137: blacklisted : /// and ones that are on probation nit: either blacklisted or on probation. http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc File be/src/scheduling/executor-blacklist.cc: http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc@51 PS5, Line 51: int64_t elapsed = MonotonicMillis() - entry.blacklist_time_ms; : entry.state = State::BLACKLISTED; : entry.blacklist_time_ms = MonotonicMillis(); : entry.cause = cause; Should MonotonicMillis() be called only once? That is, in a single time point, we update the entry. I think it would make things clear logically. http://gerrit.cloudera.org:8080/#/c/16506/5/be/src/scheduling/executor-blacklist.cc@164 PS5, Line 164: "on probation") Should we also check on NOT_BLACKLISTED here? -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 18:51:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test File testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test: http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test@278 PS3, Line 278: select * from : functional.alltypestiny a : where a.id in (select id from functional.alltypestiny limit 5) limit 5 > Right. The concern was whether the estimate is good enough for the joins. W Normally, the error becomes larger with more number of tables joined as well as the complexity of the join predicates. -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 18:33:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16522 )
Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster
..
Patch Set 1:
(3 comments)
Thanks for looking at this, it should help a lot with docker development. I
have a couple of requests to change the behaviour but I think this is a
starting point.
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh@115
PS1, Line 115: # Maximum memory available for mini-cluster and CDH cluster
I think we should call this IMPALA_CLUSTER_MAX_MEM_GB
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh@116
PS1, Line 116: export MAX_MEM_GB=28
We don't want to change the default here. Can you make the default the empty
string, then fall back to auto-detecting when it's not set.
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/start-impala-cluster.py
File bin/start-impala-cluster.py:
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/start-impala-cluster.py@574
PS1, Line 574: print("mm " + str(mem_limit))
Delete this print or replace with LOG.debug.
--
To view, visit http://gerrit.cloudera.org:8080/16522
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93
Gerrit-Change-Number: 16522
Gerrit-PatchSet: 1
Gerrit-Owner: fifteencai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 30 Sep 2020 18:30:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16291 ) Change subject: IMPALA-9180 (part 1): Remove legacy ImpalaInternalService .. Patch Set 14: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6 Gerrit-Change-Number: 16291 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 18:21:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 18:22:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10007: Impala development environment does not support Ubuntu 20.04
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16241 ) Change subject: IMPALA-10007: Impala development environment does not support Ubuntu 20.04 .. Patch Set 8: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7326/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7db302b4f1d57ec9aa2100d7589d5e814db75947 Gerrit-Change-Number: 16241 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 30 Sep 2020 18:20:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7325/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 18:20:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test File testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test: http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test@278 PS3, Line 278: select * from : functional.alltypestiny a : where a.id in (select id from functional.alltypestiny limit 5) limit 5 > would the row count estimate change after applying the small-query optimiza Right. The concern was whether the estimate is good enough for the joins. With the change, any such estimation error would be exposed. -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 18:15:26 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test File testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test: http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test@278 PS3, Line 278: select * from : functional.alltypestiny a : where a.id in (select id from functional.alltypestiny limit 5) limit 5 > May need to add other join queries to make sure the row count is estimated would the row count estimate change after applying the small-query optimization? my understanding is that this optimization is applied after single node planning which would be responsible for estimating row counts, right? -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 18:09:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10007: Impala development environment does not support Ubuntu 20.04
Qifan Chen has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/16241 ) Change subject: IMPALA-10007: Impala development environment does not support Ubuntu 20.04 .. IMPALA-10007: Impala development environment does not support Ubuntu 20.04 This is a minor amendment to a previously merged change with ChangeId I4f592f60881fd8f34e2bf393a76f5a921505010a, to address additional review comments. In particular, the original commit referred to Ubuntu 20.4 whereas it should have used Ubuntu 20.04. Change-Id: I7db302b4f1d57ec9aa2100d7589d5e814db75947 --- M bin/bootstrap_system.sh M bin/bootstrap_toolchain.py 2 files changed, 3 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/16241/8 -- To view, visit http://gerrit.cloudera.org:8080/16241 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7db302b4f1d57ec9aa2100d7589d5e814db75947 Gerrit-Change-Number: 16241 Gerrit-PatchSet: 8 Gerrit-Owner: Qifan Chen Gerrit-Reviewer: Aman Sinha Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Wenzhe Zhou has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist In current ExecuterBlacklist class, we've keyed maps on a TNetworkAddress of a backend. To simply the logic for the class, changes it to key off of the UniqueIdPB backend-id, eg. refactor 'executor_list_' to no longer be a map list> and instead makes it a map. Also fixes a minor bug with the calculation of elapsed time when a backend that was on probation is re-blacklisted. Testing: - Passed test_blacklist.py and test_query_retries.py. - Passed exhaustive tests. Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 --- M be/src/scheduling/executor-blacklist.cc M be/src/scheduling/executor-blacklist.h 2 files changed, 67 insertions(+), 103 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/16506/5 -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 5 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou
[Impala-ASF-CR] IMPALA-9711: incrementally update aggregate profile
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15931 ) Change subject: IMPALA-9711: incrementally update aggregate profile .. Patch Set 10: (1 comment) http://gerrit.cloudera.org:8080/#/c/15931/10/be/src/runtime/coordinator-backend-state.cc File be/src/runtime/coordinator-backend-state.cc: http://gerrit.cloudera.org:8080/#/c/15931/10/be/src/runtime/coordinator-backend-state.cc@539 PS10, Line 539: if (instance_stats.exec_stats_up_to_date_) continue; // Already applied. I'm thinking through this check and wondering if it interacts with finalize/done_. For V1, if this is true, then finalize = true or done_ = true, because that is the only time we call UpdateExecStats(). That sounds right to me, because if we called UpdateExecStats() with either of those set, then the completion time is set. For V2, we call UpdateExecStats() even when done_=false and finalize=false, so exec_stats_up_to_date_ can be true even though we haven't set the completion time. I think for done_ that is ok, because to transition from done_=false to done_=true involves a new status update. For finalize, there may not be a new status update, so I think we might not update the completion time correctly. -- To view, visit http://gerrit.cloudera.org:8080/15931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib03e79a40a33d8e74464640ae5f95a1467a6713a Gerrit-Change-Number: 15931 Gerrit-PatchSet: 10 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 17:37:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9180 (part 2): Refactor executor list map of ExecuterBlacklist
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/16506 ) Change subject: IMPALA-9180 (part 2): Refactor executor_list_ map of ExecuterBlacklist .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/16506/4/be/src/scheduling/executor-blacklist.h File be/src/scheduling/executor-blacklist.h: http://gerrit.cloudera.org:8080/#/c/16506/4/be/src/scheduling/executor-blacklist.h@139 PS4, Line 139: std::hash nit: I think this is unnecessary -- To view, visit http://gerrit.cloudera.org:8080/16506 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib1ae082d0e080088756af91b5b770752ca8b3aa1 Gerrit-Change-Number: 16506 Gerrit-PatchSet: 4 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 17:28:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: (1 comment) Looks good! http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test File testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test: http://gerrit.cloudera.org:8080/#/c/16521/3/testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test@278 PS3, Line 278: select * from : functional.alltypestiny a : where a.id in (select id from functional.alltypestiny limit 5) limit 5 May need to add other join queries to make sure the row count is estimated correctly. 1. cross joins 2. left/right joins -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 17:28:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6493/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 17:14:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 3 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 17:14:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9180 (part 1): Remove legacy ImpalaInternalService
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16291 ) Change subject: IMPALA-9180 (part 1): Remove legacy ImpalaInternalService .. Patch Set 14: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6492/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16291 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5fa83c8009590124dded4783f77ef70fa30119e6 Gerrit-Change-Number: 16291 Gerrit-PatchSet: 14 Gerrit-Owner: Wenzhe Zhou Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 30 Sep 2020 17:13:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 16:31:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6491/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Sep 2020 15:35:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Sep 2020 15:35:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Sep 2020 15:30:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16521 ) Change subject: IMPALA-3335: Allow single-node optimization with joins .. Patch Set 2: > The code change looks good, i think looping some queries with joins > and num_nodes=1 would be enough for me to get to a +2 Looped TPC-DS overnight with NUM_NODES=1 against an impalad mini-cluster with a single dedicated coordinator -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 15:19:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3335: Allow single-node optimization with joins
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16521 to look at the new patch set (#2). Change subject: IMPALA-3335: Allow single-node optimization with joins .. IMPALA-3335: Allow single-node optimization with joins Enable the single-node optimization for queries with joins. Testing: * Ran exhaustive tests * Looped TPC-DS overnight with NUM_NODES=1 against an impalad mini-cluster with a single dedicated coordinator Change-Id: I6b189271630214960ed482cb2b552fba9f246770 --- M fe/src/main/java/org/apache/impala/planner/Planner.java M fe/src/main/java/org/apache/impala/util/MaxRowsProcessedVisitor.java M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test 3 files changed, 21 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/21/16521/2 -- To view, visit http://gerrit.cloudera.org:8080/16521 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6b189271630214960ed482cb2b552fba9f246770 Gerrit-Change-Number: 16521 Gerrit-PatchSet: 2 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10184: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Hello Zoltan Borok-Nagy, wangsheng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16512 to look at the new patch set (#5). Change subject: IMPALA-10184: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables .. IMPALA-10184: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables A SHOW CREATE TABLE output didn't contain the PARTITION BY SPEC section for partitioned Iceberg tables. This patch addresses this shortcoming. Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test 11 files changed, 173 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16512/5 -- To view, visit http://gerrit.cloudera.org:8080/16512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 Gerrit-Change-Number: 16512 Gerrit-PatchSet: 5 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16523 ) Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7324/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 30 Sep 2020 14:31:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 21: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 21 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 14:10:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 20: Failed again with IMPALA-10143 so I uploaded a fix for that issue: https://gerrit.cloudera.org/#/c/16523/ -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 20 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 14:10:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 21: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6490/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 21 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 14:10:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10143: TestAcid.test full acid original files is flaky
Zoltan Borok-Nagy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16523 Change subject: IMPALA-10143: TestAcid.test_full_acid_original_files is flaky .. IMPALA-10143: TestAcid.test_full_acid_original_files is flaky One of the test queries depended on the values of the row__id.rowid field. In rare cases the files are written differently by Hive which leads to different 'rowid' values for the rows. I took out the test for the 'rowid' values in that particular query. We have other tests for the 'rowid' field on static data files (from 'testdata/data'), therefore we still have coverage for that and they shouldn't be flaky. Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 --- M testdata/workloads/functional-query/queries/QueryTest/full-acid-original-file.test 1 file changed, 7 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/16523/1 -- To view, visit http://gerrit.cloudera.org:8080/16523 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3d36bd23b8d3cc257bad9a83a4462f20e073d437 Gerrit-Change-Number: 16523 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-10062: TestCompressedNonText.test insensitivity to extension can fail due to wrong filename
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16518 ) Change subject: IMPALA-10062: TestCompressedNonText.test_insensitivity_to_extension can fail due to wrong filename .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I265faf8d2e7f4251b18264052eededbeb2296f57 Gerrit-Change-Number: 16518 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 13:57:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10062: TestCompressedNonText.test insensitivity to extension can fail due to wrong filename
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16518 ) Change subject: IMPALA-10062: TestCompressedNonText.test_insensitivity_to_extension can fail due to wrong filename .. IMPALA-10062: TestCompressedNonText.test_insensitivity_to_extension can fail due to wrong filename Some tests in test_compressed_formats.py hard-coded the filename of the tables. They used "00_0" for filename. The number after the underscore is the "attempt id" which can be non-zero if there were failed attempts during file writing. I modified the test to do a filesystem listing to retrieve the filename. Testing * I manually renamed one of my files to 00_1 and re-run the test. Change-Id: I265faf8d2e7f4251b18264052eededbeb2296f57 Reviewed-on: http://gerrit.cloudera.org:8080/16518 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/query_test/test_compressed_formats.py 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I265faf8d2e7f4251b18264052eededbeb2296f57 Gerrit-Change-Number: 16518 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 20: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6488/ -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 20 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 13:45:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16512 ) Change subject: IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7323/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 Gerrit-Change-Number: 16512 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 12:14:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16512 ) Change subject: IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7322/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 Gerrit-Change-Number: 16512 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 12:11:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Gabor Kaszab has posted comments on this change. (
http://gerrit.cloudera.org:8080/16512 )
Change subject: IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for
Iceberg Tables
..
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16512/1/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java
File fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java:
http://gerrit.cloudera.org:8080/#/c/16512/1/fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java@79
PS1, Line 79: public String toSql(ToSqlOptions options) {
: StringBuilder builder = new StringBuilder();
: builder.append(origFieldName_+ " " + fieldType_.toString());
: return builder.toString();
: }
:
: public TIcebergPartitionField toThrift() {
: TIcebergPartitionField result = new TIcebergPartitionField();
: result.setField_id(fieldId_);
:
> I mean can we add a new field such as 'columnName_' to record source column
Thanks for the explanation! What you're saying totally makes sense and I think
it should be part of this patch. I made the related changes to introduce a new
field for the original field name and to populate it from Iceberg.
--
To view, visit http://gerrit.cloudera.org:8080/16512
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714
Gerrit-Change-Number: 16512
Gerrit-PatchSet: 4
Gerrit-Owner: Gabor Kaszab
Gerrit-Reviewer: Gabor Kaszab
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Reviewer: wangsheng
Gerrit-Comment-Date: Wed, 30 Sep 2020 11:53:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Hello Zoltan Borok-Nagy, wangsheng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16512 to look at the new patch set (#4). Change subject: IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables .. IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables A SHOW CREATE TABLE output didn't contain the PARTITION BY SPEC section for partitioned Iceberg tables. This patch addresses this shortcoming. Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test 11 files changed, 173 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16512/4 -- To view, visit http://gerrit.cloudera.org:8080/16512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 Gerrit-Change-Number: 16512 Gerrit-PatchSet: 4 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables
Hello Zoltan Borok-Nagy, wangsheng, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16512 to look at the new patch set (#3). Change subject: IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables .. IMPALA-10187: Add PARTITON BY SPEC to SHOW CREATE TABLE for Iceberg Tables A SHOW CREATE TABLE output didn't contain the PARTITION BY SPEC section for partitioned Iceberg tables. This patch addresses this shortcoming. Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 --- M common/thrift/CatalogObjects.thrift M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionField.java M fe/src/main/java/org/apache/impala/analysis/IcebergPartitionSpec.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M testdata/workloads/functional-query/queries/QueryTest/show-create-table.test 11 files changed, 174 insertions(+), 33 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/12/16512/3 -- To view, visit http://gerrit.cloudera.org:8080/16512 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie4c43b75057807ab513a220d348155be2487e714 Gerrit-Change-Number: 16512 Gerrit-PatchSet: 3 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng
[Impala-ASF-CR] IMPALA-10062: TestCompressedNonText.test insensitivity to extension can fail due to wrong filename
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16518 ) Change subject: IMPALA-10062: TestCompressedNonText.test_insensitivity_to_extension can fail due to wrong filename .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I265faf8d2e7f4251b18264052eededbeb2296f57 Gerrit-Change-Number: 16518 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 08:39:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10062: TestCompressedNonText.test insensitivity to extension can fail due to wrong filename
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16518 ) Change subject: IMPALA-10062: TestCompressedNonText.test_insensitivity_to_extension can fail due to wrong filename .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6489/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16518 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I265faf8d2e7f4251b18264052eededbeb2296f57 Gerrit-Change-Number: 16518 Gerrit-PatchSet: 2 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 08:39:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 20: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6488/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 20 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 08:21:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10164: Supporting HadoopCatalog for Iceberg table
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/16446 ) Change subject: IMPALA-10164: Supporting HadoopCatalog for Iceberg table .. Patch Set 20: Code-Review+2 Those XFAIL failures can be omitted, not those the ones that make the build fail. However, there is a failing test in this job: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/12194/ I see test_full_acid_original_files failing but that seems to be flaky anyway so I'm sure it's unrelated to this change. https://issues.apache.org/jira/browse/IMPALA-10143 Let me re-run the verify job. Carry +2 from Zoltan. -- To view, visit http://gerrit.cloudera.org:8080/16446 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic1893c50a633ca22d4bca6726c9937b026f5d5ef Gerrit-Change-Number: 16446 Gerrit-PatchSet: 20 Gerrit-Owner: wangsheng Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Reviewer: wangsheng Gerrit-Comment-Date: Wed, 30 Sep 2020 08:20:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7321/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 1 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 08:17:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16522 Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed in containers whose mem limit are lower than the host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra memories. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left memory into threasholds. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable MAX_MEM_GB in 'bin/impala-config.sh' - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking MAX_MEM_GB rather than sys_mem into account. - Modified the algorightm in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py', Similarly, making MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/1 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 1 Gerrit-Owner: fifteencai
