[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224476#comment-14224476
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-------------------------------------------------------------
Commit bb81082e58d90da9971b7e06f4c3f8639446b2d2 in cloudstack's branch
refs/heads/4.3 from [~rohit.ya...@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=bb81082 ]
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
The -Xms value specifies the minimum heap size the JVM should start with and
-Xmx is the maximum heap size it can grow. The previous fix imposed minimum
limit of 1G which is unreasonably for small deployments. The fix is to start
with 256MB and limit to 2G for cloudstack-agent process. This was tested on
DevCloud/KVM and then again on a ACS/KVM deployment on real hardware.
With these values, it's possible for the agent to work in a DevCloud/KVM
environment and if JVM needs it can increase the heap size to 2G. The fix also
ports these settings to Debian cloud-agent init.d script as well.
Signed-off-by: Rohit Yadav <rohit.ya...@shapeblue.com>
> cloudstack-agent jsvc gets too large virtual memory space.
> ----------------------------------------------------------
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
> Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> Reporter: Keiichi Yusa
> Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> - $JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> + $JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
