[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed when running in docker containers with a lower mem limit than host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra resource. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left into threasholds below. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account. - Modified the logic in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py' Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Reviewed-on: http://gerrit.cloudera.org:8080/16522 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 9 insertions(+), 3 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 5 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 4 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 08:38:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7330/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:35:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6494/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 4 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 4 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7329/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 2 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 01 Oct 2020 03:22:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed when running in docker containers with a lower mem limit than host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra resource. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left into threasholds below. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account. - Modified the logic in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py' Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 9 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/3 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 3 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed in containers whose mem limit are lower than the host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra memories. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left memory into threasholds. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable IMPALA_CLUSTER_MAX_MEM_GB - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking IMPALA_CLUSTER_MAX_MEM_GB rather than sys_mem into account. - Modified the logic in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py' Similarly, making IMPALA_CLUSTER_MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/2 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 2 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/16522 )
Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster
..
Patch Set 1:
(3 comments)
Thanks for looking at this, it should help a lot with docker development. I
have a couple of requests to change the behaviour but I think this is a
starting point.
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh@115
PS1, Line 115: # Maximum memory available for mini-cluster and CDH cluster
I think we should call this IMPALA_CLUSTER_MAX_MEM_GB
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/impala-config.sh@116
PS1, Line 116: export MAX_MEM_GB=28
We don't want to change the default here. Can you make the default the empty
string, then fall back to auto-detecting when it's not set.
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/start-impala-cluster.py
File bin/start-impala-cluster.py:
http://gerrit.cloudera.org:8080/#/c/16522/1/bin/start-impala-cluster.py@574
PS1, Line 574: print("mm " + str(mem_limit))
Delete this print or replace with LOG.debug.
--
To view, visit http://gerrit.cloudera.org:8080/16522
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93
Gerrit-Change-Number: 16522
Gerrit-PatchSet: 1
Gerrit-Owner: fifteencai
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Wed, 30 Sep 2020 18:30:11 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16522 ) Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7321/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 1 Gerrit-Owner: fifteencai Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 30 Sep 2020 08:17:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10193: Limit the memory usage for the whole test cluster
fifteencai has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16522 Change subject: IMPALA-10193: Limit the memory usage for the whole test cluster .. IMPALA-10193: Limit the memory usage for the whole test cluster This patch introduces a new approach of limiting the memory usage for both mini-cluster and CDH cluster. Without this limit, clusters are prone to getting killed in containers whose mem limit are lower than the host's memory size. i.e. The mini-cluster may running in a container with 32GB limitted by CGROUPS, while the host machine has 128GB. Under this circumstance, if the container is started with '-privileged' command argument, both mini and CDH clusters compute their mem_limit according to 128GB rather than 32GB. They will be killed when attempting to apply for extra memories. Currently, the mem-limit estimating algorithms for Impalad and Node Manager are different: for Impalad: mem_limit = 0.7 * sys_mem / cluster_size (default is 3) for Node Manager: 1. Leave aside 24GB, then fit the left memory into threasholds. 2. The bare limit is 4GB and maximum limit 48GB In headge of over-consumption, we - Added a new environment variable MAX_MEM_GB in 'bin/impala-config.sh' - Modified the algorithm in 'bin/start-impala-cluster.py', making it taking MAX_MEM_GB rather than sys_mem into account. - Modified the algorightm in 'testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py', Similarly, making MAX_MEM_GB substitutes for sys_mem . Testing: this patch worked in a 32GB docker container running on a 128GB host machine. All 1188 unit tests get passed. Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.py 3 files changed, 10 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/16522/1 -- To view, visit http://gerrit.cloudera.org:8080/16522 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8537fd748e279d5a0e689872aeb4dbfd0c84dc93 Gerrit-Change-Number: 16522 Gerrit-PatchSet: 1 Gerrit-Owner: fifteencai
