[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356651#comment-14356651
]
ASF GitHub Bot commented on CLOUDSTACK-7951:
GitHub user ghxandsky opened a pull request:
https://github.com/apache/cloudstack/pull/111
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
According https://issues.apache.org/jira/browse/CLOUDSTACK-7951, and
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=207d465, also fix to
centos7.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ghxandsky/cloudstack master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/111.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #111
commit 0b01343a1af24441d338bc04492de1dce5d34b0a
Author: Star Guo
Date: 2015-03-11T09:50:39Z
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Assignee: Rohit Yadav
>Priority: Critical
> Fix For: Future, 4.5.0, 4.6.0, 4.4.3, 4.3.2
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356780#comment-14356780
]
ASF GitHub Bot commented on CLOUDSTACK-7951:
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/111
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Assignee: Rohit Yadav
>Priority: Critical
> Fix For: Future, 4.5.0, 4.6.0, 4.4.3, 4.3.2
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356728#comment-14356728
]
ASF GitHub Bot commented on CLOUDSTACK-7951:
Github user bhaisaab commented on the pull request:
https://github.com/apache/cloudstack/pull/111#issuecomment-78244680
LGTM!
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Assignee: Rohit Yadav
>Priority: Critical
> Fix For: Future, 4.5.0, 4.6.0, 4.4.3, 4.3.2
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14356660#comment-14356660
]
ASF GitHub Bot commented on CLOUDSTACK-7951:
Github user ghxandsky commented on the pull request:
https://github.com/apache/cloudstack/pull/111#issuecomment-78234938
I have checked this issue in fedora20 & fedora21 is fixed.
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Assignee: Rohit Yadav
>Priority: Critical
> Fix For: Future, 4.5.0, 4.6.0, 4.4.3, 4.3.2
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225164#comment-14225164
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit bb36ef248fe761fb56068cffd592936bd811 in cloudstack's branch
refs/heads/4.4 from [~yusa]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=bb36ef2 ]
CLOUDSTACK-7951
Limit amount of memory used by cloudstack-agent jsvc.
Signed-off-by: Edison Su
(cherry picked from commit 1d6ca5eacb1725cc8485c4564aaf7332682ab5b2)
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225165#comment-14225165
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 207d4653fa4ba9647ec4783faca06430a182e87b in cloudstack's branch
refs/heads/4.4 from [~rohit.ya...@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=207d465 ]
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
The -Xms value specifies the minimum heap size the JVM should start with and
-Xmx is the maximum heap size it can grow. The previous fix imposed minimum
limit of 1G which is unreasonably for small deployments. The fix is to start
with 256MB and limit to 2G for cloudstack-agent process. This was tested on
DevCloud/KVM and then again on a ACS/KVM deployment on real hardware.
With these values, it's possible for the agent to work in a DevCloud/KVM
environment and if JVM needs it can increase the heap size to 2G. The fix also
ports these settings to Debian cloud-agent init.d script as well.
Signed-off-by: Rohit Yadav
(cherry picked from commit bb81082e58d90da9971b7e06f4c3f8639446b2d2)
Signed-off-by: Rohit Yadav
(cherry picked from commit 3fe1f732d73d9ce12837d48d15031bc41df46b4d)
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224496#comment-14224496
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 19d73fe67d6c43ec4ef7c4a8e76c09e15e05a830 in cloudstack's branch
refs/heads/master from [~rohit.ya...@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=19d73fe ]
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
The -Xms value specifies the minimum heap size the JVM should start with and
-Xmx is the maximum heap size it can grow. The previous fix imposed minimum
limit of 1G which is unreasonably for small deployments. The fix is to start
with 256MB and limit to 2G for cloudstack-agent process. This was tested on
DevCloud/KVM and then again on a ACS/KVM deployment on real hardware.
With these values, it's possible for the agent to work in a DevCloud/KVM
environment and if JVM needs it can increase the heap size to 2G. The fix also
ports these settings to Debian cloud-agent init.d script as well.
Signed-off-by: Rohit Yadav
(cherry picked from commit bb81082e58d90da9971b7e06f4c3f8639446b2d2)
Signed-off-by: Rohit Yadav
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224479#comment-14224479
]
Rohit Yadav commented on CLOUDSTACK-7951:
-
[~dahn] do you want to cherry-pick this?
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224477#comment-14224477
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 3fe1f732d73d9ce12837d48d15031bc41df46b4d in cloudstack's branch
refs/heads/4.5 from [~rohit.ya...@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=3fe1f73 ]
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
The -Xms value specifies the minimum heap size the JVM should start with and
-Xmx is the maximum heap size it can grow. The previous fix imposed minimum
limit of 1G which is unreasonably for small deployments. The fix is to start
with 256MB and limit to 2G for cloudstack-agent process. This was tested on
DevCloud/KVM and then again on a ACS/KVM deployment on real hardware.
With these values, it's possible for the agent to work in a DevCloud/KVM
environment and if JVM needs it can increase the heap size to 2G. The fix also
ports these settings to Debian cloud-agent init.d script as well.
Signed-off-by: Rohit Yadav
(cherry picked from commit bb81082e58d90da9971b7e06f4c3f8639446b2d2)
Signed-off-by: Rohit Yadav
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224476#comment-14224476
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit bb81082e58d90da9971b7e06f4c3f8639446b2d2 in cloudstack's branch
refs/heads/4.3 from [~rohit.ya...@shapeblue.com]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=bb81082 ]
CLOUDSTACK-7951: Limit amount of memory used by cloudstack-agent jsvc
The -Xms value specifies the minimum heap size the JVM should start with and
-Xmx is the maximum heap size it can grow. The previous fix imposed minimum
limit of 1G which is unreasonably for small deployments. The fix is to start
with 256MB and limit to 2G for cloudstack-agent process. This was tested on
DevCloud/KVM and then again on a ACS/KVM deployment on real hardware.
With these values, it's possible for the agent to work in a DevCloud/KVM
environment and if JVM needs it can increase the heap size to 2G. The fix also
ports these settings to Debian cloud-agent init.d script as well.
Signed-off-by: Rohit Yadav
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224475#comment-14224475
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 20de43fd792390d3ee57a3006d937413931ee371 in cloudstack's branch
refs/heads/4.3 from [~yusa]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=20de43f ]
CLOUDSTACK-7951
Limit amount of memory used by cloudstack-agent jsvc.
Signed-off-by: Edison Su
(cherry picked from commit 7884c750a205fa8962de4ac65960a9f514770817)
Signed-off-by: Rohit Yadav
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14221941#comment-14221941
]
Rohit Yadav commented on CLOUDSTACK-7951:
-
[~edison]
There is a (potential) issue with the mix; the Xms value is 1GB that means the
initial heap size for JVM is that value (this fails for people using DevCloud
etc.); and the maximum heap size Xmx is set to 2GB so mgmt server may never
use/exploit more memory even if the server has some. Additional memory is
useful for a lot of dynamically resizable internal data-structures such as
temporary data structures, maps and lists so should we impose such limits?
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
> Fix For: Future, 4.3.0, 4.4.0, 4.5.0, 4.6.0
>
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219706#comment-14219706
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 7884c750a205fa8962de4ac65960a9f514770817 in cloudstack's branch
refs/heads/master from [~yusa]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=7884c75 ]
CLOUDSTACK-7951
Limit amount of memory used by cloudstack-agent jsvc.
Signed-off-by: Edison Su
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219705#comment-14219705
]
ASF subversion and git services commented on CLOUDSTACK-7951:
-
Commit 1d6ca5eacb1725cc8485c4564aaf7332682ab5b2 in cloudstack's branch
refs/heads/4.5 from [~yusa]
[ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=1d6ca5e ]
CLOUDSTACK-7951
Limit amount of memory used by cloudstack-agent jsvc.
Signed-off-by: Edison Su
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CLOUDSTACK-7951) cloudstack-agent jsvc gets too large virtual memory space.
[
https://issues.apache.org/jira/browse/CLOUDSTACK-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219178#comment-14219178
]
Keiichi Yusa commented on CLOUDSTACK-7951:
--
This is a patch: https://reviews.apache.org/r/28279/
> cloudstack-agent jsvc gets too large virtual memory space.
> --
>
> Key: CLOUDSTACK-7951
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7951
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM
>Affects Versions: 4.3.0
> Environment: - Builds up with 155 physical machines and a cloudstack
> server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads).
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
>Reporter: Keiichi Yusa
>Priority: Critical
>
> cloudstack-agent jsvc gets too large virtual memory space on huge
> memory equipped machine.
> We have 155 physical machines that each is equipped 128GB RAM and
> is installed CentOS6.6 (64bit). On these physical machines, I use
> KVM as hypervisor.
> I create Compute Offering with 120GB RAM and I deploy VM with this
> Compute Offering on this environment.
> Now, I have a problem that qemu-kvm process often fails deploying VM
> in the above conditions. I confirm that the following message is
> outputted in /var/log/libvirt/qemu/i-X-X-VM.log.
> {noformat}
> "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> {noformat}
> I find that cloudstack-agent jsvc gets virtual memory about 35GB.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 5787 root 20 0 34.9g 667m 18m S 0.0 0.5 8:45.20 jsvc
> (snip)
> {noformat}
>
> I suspect that this failure is caused by what cloudstack-agent jsvc
> gets too large virtual memory.
> I try to patch /etc/init.d/cloudstack-agent as follows:
> {noformat}
> @@ -66,7 +66,7 @@
> start() {
> echo -n $"Starting $PROGNAME: "
> if hostname --fqdn >/dev/null 2>&1 ; then
> -$JSVC -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> +$JSVC -Xms1024m -Xmx2048m -cp "$CLASSPATH" -pidfile "$PIDFILE" \
> -errfile $LOGDIR/cloudstack-agent.err -outfile
> $LOGDIR/cloudstack-agent.out $CLASS
> RETVAL=$?
> echo
> {noformat}
> Then, I restart cloudstack-agent.
> As a result, cloudstack agent jsvc reduces virtual memory about 6GB.
> Also, qemu-kvm process does not fail deploying VM for now.
> {noformat}
> $ top -a
> (snip)
>PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
> 141405 root 20 0 6290m 393m 18m S 0.0 0.3 1:11.44 jsvc
> (snip)
> {noformat}
> If that helps, our CloudStack environment is as follows:
> {noformat}
> - Builds up with 155 physical machines and a cloudstack server.
> - Each physical machine equips these.
> + Xeon E5-2680 (10 cores, 20 threads)
> + 128GB memory (and is not set swap space).
> + 600GB Solid State Drive.
> - All physical machines are networked by InfiniBand and 10GBase-T.
> - Using CentOS6.6 (64bit)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
