[ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Hello All, I am using centos6.5 x64 on a server with 48 G RAM and 8 Cores.Managed by Ovirt There is only one running VM with RAM 34 G and with 6 VCPU (pinned to proper numa nodes) from top top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, 0.17 Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers Swap: 12337144k total,0k used, 12337144k free, 343052k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 34096 -realtime mlock=off -smp 6,maxcpus=160,sockets=80,c from /sys/kernel/mm/ksm pages_unshared 7602322 pages_shared 207023 pages_to_scan 64 pages_volatile31678 Any idea why ksmd is not coming to normal CPU usage ,on a different server ksmd was disabled and for testing when i enabled it initially CPU usage was high but later settled down to 3% ,in that host i have 4 VMs running. Before turning off ksmd can any one help me find out why ksmd is behaving like this.Initially it had 2 virtual machines,because of high CPU utilization of this guest other is migrated to another host. Thanks ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Memory usage > 80%: ksm kicks in. There it will run at full speed until usage is below 80%. There is an open BZ from me. Bad behaviour is controlled by mom. Markus Am 06.12.2014 15:58 schrieb mad Engineer : Hello All, I am using centos6.5 x64 on a server with 48 G RAM and 8 Cores.Managed by Ovirt There is only one running VM with RAM 34 G and with 6 VCPU (pinned to proper numa nodes) from top top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, 0.17 Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers Swap: 12337144k total,0k used, 12337144k free, 343052k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem -enable-kvm -m 34096 -realtime mlock=off -smp 6,maxcpus=160,sockets=80,c from /sys/kernel/mm/ksm pages_unshared 7602322 pages_shared 207023 pages_to_scan 64 pages_volatile31678 Any idea why ksmd is not coming to normal CPU usage ,on a different server ksmd was disabled and for testing when i enabled it initially CPU usage was high but later settled down to 3% ,in that host i have 4 VMs running. Before turning off ksmd can any one help me find out why ksmd is behaving like this.Initially it had 2 virtual machines,because of high CPU utilization of this guest other is migrated to another host. Thanks ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Ãber das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Thanks for the info Markus free -g total used free sharedbuffers cached Mem:47 32 14 0 0 0 % of usage is 68 . Why is ksmd not sleeping.Do you have any idea? regards, On Sat, Dec 6, 2014 at 8:45 PM, Markus Stockhausen wrote: > Memory usage > 80%: ksm kicks in. There it will run at full speed until > usage is below 80%. There is an open BZ from me. Bad behaviour is controlled > by mom. > > Markus > > Am 06.12.2014 15:58 schrieb mad Engineer : > > Hello All, > I am using centos6.5 x64 on a server with 48 G RAM and 8 > Cores.Managed by Ovirt > There is only one running VM with RAM 34 G and with 6 VCPU (pinned to > proper numa nodes) > > from top > > top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, 0.17 > Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie > Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers > Swap: 12337144k total,0k used, 12337144k free, 343052k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] > 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 > /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid > 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 > /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem > -enable-kvm -m 34096 -realtime mlock=off -smp > 6,maxcpus=160,sockets=80,c > > from /sys/kernel/mm/ksm > pages_unshared 7602322 > pages_shared 207023 > pages_to_scan 64 > pages_volatile31678 > > Any idea why ksmd is not coming to normal CPU usage ,on a different > server ksmd was disabled and for testing when i enabled it initially > CPU usage was high but later settled down to 3% ,in that host i have 4 > VMs running. > > Before turning off ksmd can any one help me find out why ksmd is > behaving like this.Initially it had 2 virtual machines,because of high > CPU utilization of this guest other is migrated to another host. > > Thanks > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Hi, you can see all the relevant definitions in mom's ksm policy. The trigger to run ksm is defined here: https://github.com/oVirt/vdsm/blob/master/vdsm/mom.d/03-ksm.policy#L23 You can change this setting (in each host you have) to something that suits your load. We have the below bug opened and hopefully handle it in one of the next versions. https://bugzilla.redhat.com/show_bug.cgi?id=1026294 Doron - Original Message - > From: "mad Engineer" > To: "Markus Stockhausen" > Cc: "users" > Sent: Saturday, December 6, 2014 6:13:20 PM > Subject: Re: [ovirt-users] ksmd high cpu usage from almost a week with just > one vm running > > Thanks for the info Markus > > free -g > total used free sharedbuffers cached > Mem:47 32 14 0 0 0 > > % of usage is 68 . > > Why is ksmd not sleeping.Do you have any idea? > regards, > > > On Sat, Dec 6, 2014 at 8:45 PM, Markus Stockhausen > wrote: > > Memory usage > 80%: ksm kicks in. There it will run at full speed until > > usage is below 80%. There is an open BZ from me. Bad behaviour is > > controlled > > by mom. > > > > Markus > > > > Am 06.12.2014 15:58 schrieb mad Engineer : > > > > Hello All, > > I am using centos6.5 x64 on a server with 48 G RAM and 8 > > Cores.Managed by Ovirt > > There is only one running VM with RAM 34 G and with 6 VCPU (pinned to > > proper numa nodes) > > > > from top > > > > top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, 0.17 > > Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie > > Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers > > Swap: 12337144k total,0k used, 12337144k free, 343052k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > > 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] > > 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 > > /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid > > 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 > > /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem > > -enable-kvm -m 34096 -realtime mlock=off -smp > > 6,maxcpus=160,sockets=80,c > > > > from /sys/kernel/mm/ksm > > pages_unshared 7602322 > > pages_shared 207023 > > pages_to_scan 64 > > pages_volatile31678 > > > > Any idea why ksmd is not coming to normal CPU usage ,on a different > > server ksmd was disabled and for testing when i enabled it initially > > CPU usage was high but later settled down to 3% ,in that host i have 4 > > VMs running. > > > > Before turning off ksmd can any one help me find out why ksmd is > > behaving like this.Initially it had 2 virtual machines,because of high > > CPU utilization of this guest other is migrated to another host. > > > > Thanks > > ___ > > Users mailing list > > Users@ovirt.org > > http://lists.ovirt.org/mailman/listinfo/users > ___ > Users mailing list > Users@ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Thanks Doron, Can you confirm this behaviour,i changed (defvar ksm_free_percent 0.10) and restarted vdsm and now ksmd is behaving normally.But i don't understand why its behaving aggressively when free RAM percentages is 22% even though its close to 20 :) Regards On Sun, Dec 7, 2014 at 6:13 PM, Doron Fediuck wrote: > Hi, > you can see all the relevant definitions in mom's ksm policy. > The trigger to run ksm is defined here: > https://github.com/oVirt/vdsm/blob/master/vdsm/mom.d/03-ksm.policy#L23 > > You can change this setting (in each host you have) to something that > suits your load. We have the below bug opened and hopefully handle it > in one of the next versions. > https://bugzilla.redhat.com/show_bug.cgi?id=1026294 > > Doron > > - Original Message - >> From: "mad Engineer" >> To: "Markus Stockhausen" >> Cc: "users" >> Sent: Saturday, December 6, 2014 6:13:20 PM >> Subject: Re: [ovirt-users] ksmd high cpu usage from almost a week with just >> one vm running >> >> Thanks for the info Markus >> >> free -g >> total used free sharedbuffers cached >> Mem:47 32 14 0 0 0 >> >> % of usage is 68 . >> >> Why is ksmd not sleeping.Do you have any idea? >> regards, >> >> >> On Sat, Dec 6, 2014 at 8:45 PM, Markus Stockhausen >> wrote: >> > Memory usage > 80%: ksm kicks in. There it will run at full speed until >> > usage is below 80%. There is an open BZ from me. Bad behaviour is >> > controlled >> > by mom. >> > >> > Markus >> > >> > Am 06.12.2014 15:58 schrieb mad Engineer : >> > >> > Hello All, >> > I am using centos6.5 x64 on a server with 48 G RAM and 8 >> > Cores.Managed by Ovirt >> > There is only one running VM with RAM 34 G and with 6 VCPU (pinned to >> > proper numa nodes) >> > >> > from top >> > >> > top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, 0.17 >> > Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie >> > Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, >> > 0.0%st >> > Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers >> > Swap: 12337144k total,0k used, 12337144k free, 343052k cached >> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND >> > 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] >> > 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 >> > /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid >> > 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 >> > /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem >> > -enable-kvm -m 34096 -realtime mlock=off -smp >> > 6,maxcpus=160,sockets=80,c >> > >> > from /sys/kernel/mm/ksm >> > pages_unshared 7602322 >> > pages_shared 207023 >> > pages_to_scan 64 >> > pages_volatile31678 >> > >> > Any idea why ksmd is not coming to normal CPU usage ,on a different >> > server ksmd was disabled and for testing when i enabled it initially >> > CPU usage was high but later settled down to 3% ,in that host i have 4 >> > VMs running. >> > >> > Before turning off ksmd can any one help me find out why ksmd is >> > behaving like this.Initially it had 2 virtual machines,because of high >> > CPU utilization of this guest other is migrated to another host. >> > >> > Thanks >> > ___ >> > Users mailing list >> > Users@ovirt.org >> > http://lists.ovirt.org/mailman/listinfo/users >> ___ >> Users mailing list >> Users@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] ksmd high cpu usage from almost a week with just one vm running
Hi, First of all you need to remember this is accounting all available memory cache included, so what you see as 22 may not be what mom is using. Additionally some of the calculations take time for ksm, so it takes time for everything to stabilize once you hit a peak. - Original Message - > From: "mad Engineer" gmail.com> > To: "Doron Fediuck" > Cc: "Markus Stockhausen" , "users" ovirt.org>, > "Martin Sivak" , > "Roy Golan" > Sent: Sunday, December 7, 2014 6:52:16 PM > Subject: Re: [ovirt-users] ksmd high cpu usage from almost a week with just > one vm running > > Thanks Doron, > Can you confirm this behaviour,i changed (defvar > ksm_free_percent 0.10) and restarted vdsm and now ksmd is behaving > normally.But i don't understand why its behaving aggressively when > free RAM percentages is 22% even though its close to 20 :) > > Regards > > On Sun, Dec 7, 2014 at 6:13 PM, Doron Fediuck wrote: > > Hi, > > you can see all the relevant definitions in mom's ksm policy. > > The trigger to run ksm is defined here: > > https://github.com/oVirt/vdsm/blob/master/vdsm/mom.d/03-ksm.policy#L23 > > > > You can change this setting (in each host you have) to something that > > suits your load. We have the below bug opened and hopefully handle it > > in one of the next versions. > > https://bugzilla.redhat.com/show_bug.cgi?id=1026294 > > > > Doron > > > > ----- Original Message - > >> From: "mad Engineer" gmail.com> > >> To: "Markus Stockhausen" > >> Cc: "users" ovirt.org> > >> Sent: Saturday, December 6, 2014 6:13:20 PM > >> Subject: Re: [ovirt-users] ksmd high cpu usage from almost a week with > >> just one vm running > >> > >> Thanks for the info Markus > >> > >> free -g > >> total used free sharedbuffers cached > >> Mem:47 32 14 0 0 0 > >> > >> % of usage is 68 . > >> > >> Why is ksmd not sleeping.Do you have any idea? > >> regards, > >> > >> > >> On Sat, Dec 6, 2014 at 8:45 PM, Markus Stockhausen > >> wrote: > >> > Memory usage > 80%: ksm kicks in. There it will run at full speed until > >> > usage is below 80%. There is an open BZ from me. Bad behaviour is > >> > controlled > >> > by mom. > >> > > >> > Markus > >> > > >> > Am 06.12.2014 15:58 schrieb mad Engineer gmail.com>: > >> > > >> > Hello All, > >> > I am using centos6.5 x64 on a server with 48 G RAM and 8 > >> > Cores.Managed by Ovirt > >> > There is only one running VM with RAM 34 G and with 6 VCPU (pinned to > >> > proper numa nodes) > >> > > >> > from top > >> > > >> > top - 06:42:48 up 67 days, 20:05, 1 user, load average: 0.26, 0.20, > >> > 0.17 > >> > Tasks: 285 total, 2 running, 282 sleeping, 0 stopped, 1 zombie > >> > Cpu(s): 1.0%us, 1.4%sy, 0.0%ni, 97.5%id, 0.1%wa, 0.0%hi, 0.0%si, > >> > 0.0%st > >> > Mem: 49356468k total, 33977684k used, 15378784k free, 142812k buffers > >> > Swap: 12337144k total,0k used, 12337144k free, 343052k cached > >> > > >> > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > >> > 101 root 25 5 000 R 27.4 0.0 5650:04 [ksmd] > >> > 26004 vdsm 0 -20 3371m 64m 9400 S 9.8 0.1 1653:27 > >> > /usr/bin/python /usr/share/vdsm/vdsm --pidfile /var/run/vdsm/vdsmd.pid > >> > 20963 qemu 20 0 38.5g 33g 6792 S 3.9 71.6 5225:43 > >> > /usr/libexec/qemu-kvm -name Cinder -S -M rhel6.5.0 -cpu Nehalem > >> > -enable-kvm -m 34096 -realtime mlock=off -smp > >> > 6,maxcpus=160,sockets=80,c > >> > > >> > from /sys/kernel/mm/ksm > >> > pages_unshared 7602322 > >> > pages_shared 207023 > >> > pages_to_scan 64 > >> > pages_volatile31678 > >> > > >> > Any idea why ksmd is not coming to normal CPU usage ,on a different > >> > server ksmd was disabled and for testing when i enabled it initially > >> > CPU usage was high but later settled down to 3% ,in that host i have 4 > >> > VMs running. > >> > > >> > Before turning off ksmd can any one help me find out why ksmd is > >> > behaving like this.Initially it had 2 virtual machines,because of high > >> > CPU utilization of this guest other is migrated to another host. > >> > > >> > Thanks > >> > ___ > >> > Users mailing list > >> > Users@ovirt.org > >> > http://lists.ovirt.org/mailman/listinfo/users > >> ___ > >> Users mailing list > >> Users@ovirt.org > >> http://lists.ovirt.org/mailman/listinfo/users > >> > ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
