Re: KVM slow LAMP guest
On 8/26/15 6:41 AM, Hansa wrote: On 26-8-2015 0:33, Wanpeng Li wrote: On the VM server I issued the command below every eleven minutes: date >> curltest-file; _ top -b -n 1 | sed -n '7,12p' >> curltest-file; _ curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file. I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar) issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total. If you just use halt_poll_ns or both halt_poll_ns and idle=poll in guest? I just use kvm.halt_poll_ns=50 Should I try some different tests? Looks good to me currently. Per vCPU will consume almost half pCPU's capacity in host when add idle=poll in my testing which is not suitable for some cloud computing scenarios since vCPUs have high overcommit ratio on host. Regards, Wanpeng Li -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 26-8-2015 0:33, Wanpeng Li wrote: On the VM server I issued the command below every eleven minutes: date >> curltest-file; _ top -b -n 1 | sed -n '7,12p' >> curltest-file; _ curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file. I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar) issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total. If you just use halt_poll_ns or both halt_poll_ns and idle=poll in guest? I just use kvm.halt_poll_ns=50 Should I try some different tests? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 8/25/15 11:42 PM, Hansa wrote: On 24-8-2015 1:26, Wanpeng Li wrote: On 8/24/15 3:18 AM, Hansa wrote: On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "C. Bröcker" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Hi, I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available. Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem! How much seconds to load your Wordpress site this time? Regards, Wanpeng Li The average is around 0.4 seconds to load my "heaviest" site on my slowest machine. Nice! On the VM server I issued the command below every eleven minutes: date >> curltest-file; _ top -b -n 1 | sed -n '7,12p' >> curltest-file; _ curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file. I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar) issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total. If you just use halt_poll_ns or both halt_poll_ns and idle=poll in guest? Regards, Wanpeng Li Cheers and good luck tweaking your sites! Best, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 24-8-2015 1:26, Wanpeng Li wrote: On 8/24/15 3:18 AM, Hansa wrote: On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "C. Bröcker" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Hi, I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available. Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem! How much seconds to load your Wordpress site this time? Regards, Wanpeng Li The average is around 0.4 seconds to load my "heaviest" site on my slowest machine. On the VM server I issued the command below every eleven minutes: date >> curltest-file; _ top -b -n 1 | sed -n '7,12p' >> curltest-file; _ curl -o /dev/null -s -w"time_total: %{time_total}\\n" https://my.domain.com | perl -pe 'BEGIN {use POSIX;} print strftime("%Y-%m-%d %H:%M:%S ", localtime)' >> curltest-file This gives me the total time for displaying my site on a local machine. It also includes a 'top' command to display which processes are running at each sample. All is saved in a file called curltest-file. I found 7 occurrences in my curltest-file of a time_total larger than 20 seconds. Top however didn't show any significant CPU or IO activity at those sampled times. Further investigations shows me that they are related to a known (gravatar) issue in the Wordpress Jetpack plugin. I didn't include these samples in the average total. Cheers and good luck tweaking your sites! Best, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 24-8-2015 1:26, Wanpeng Li wrote: On 8/24/15 3:18 AM, Hansa wrote: On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "C. Bröcker" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Hi, I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available. Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem! How much seconds to load your Wordpress site this time? Regards, Wanpeng Li I've only tested availability using the webmin's "Remote HTTP Service" monitoring tool. Currently it only times out when doing backups. I will do some tests with curl and let you know the result in a day or so. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 8/24/15 3:18 AM, Hansa wrote: On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "C. Bröcker" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Hi, I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available. Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem! How much seconds to load your Wordpress site this time? Regards, Wanpeng Li My guess is that a lot people out there have the same problem but do not know the solution. Why not put it in the kernel by default? Thanks a lot for all the help! - Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "C. Bröcker" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Hi, I upgraded to the CentOS 6.7 release which came out last month and as promised the halt_poll_ns parameter was available. Last week I tested the availability status every 5 minutes on my Wordpress VM's with the halt_poll_ns kernel param set on DOM0. I'm pleased to announce that it solves the problem! My guess is that a lot people out there have the same problem but do not know the solution. Why not put it in the kernel by default? Thanks a lot for all the help! - Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 16-7-2015 13:27, Paolo Bonzini wrote: On 15/07/2015 22:02, "Hansa" wrote: What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Thanks Paolo! I'll await the 6.7 release. Best, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 15/07/2015 22:02, "C. Bröcker" wrote: >> >> What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 >> which will be out in a few days/weeks. >> >> Paolo > OK. As said CentOS 6.6. > But where do I put this parameter? You can add "kvm.halt_poll_ns=50" to the kernel command line. If you have the parameter, you have the /sys/module/kvm/parameters/halt_poll_ns file. Paolo > Regards, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 14-7-2015 23:16, Paolo Bonzini wrote: On 14/07/2015 21:54, "Hansa" wrote: [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926 Sorry David. That is not an option. It's a production server on 2.6 kernel. 4.0 kernel is not in Base repos and I don't know which impact installing the newer kernel version will have on this server. Any other options? You could backport the patch to the version you have. What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo OK. As said CentOS 6.6. But where do I put this parameter? Regards, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 14-7-2015 23:16, Paolo Bonzini wrote: On 14/07/2015 21:54, "Hansa" wrote: [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926 Sorry David. That is not an option. It's a production server on 2.6 kernel. 4.0 kernel is not in Base repos and I don't know which impact installing the newer kernel version will have on this server. Any other options? You could backport the patch to the version you have. What OS is this? Is it RHEL/CentOS? If so, halt_poll_ns will be in 6.7 which will be out in a few days/weeks. Paolo CentOS release 6.6 (Final) -Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On 14-7-2015 19:16, David Matlack wrote: (adding back the kvm mailing list) On Tue, Jul 14, 2015 at 1:12 AM, "C. Bröcker" wrote: On 14-7-2015 10:04, Hansa wrote: On 13-7-2015 20:57, David Matlack wrote: On Thu, Jun 18, 2015 at 10:26 AM, David Matlack wrote: On Thu, Jun 18, 2015 at 1:25 AM, Hansa wrote: Hi, I have a LAMP server as guest in KVM. Whenever the server is idle for some time it takes about 30 seconds to load a Wordpress site. If the server is not idle the site shows up in max 5 seconds. I've already turned of power management in the guest by passing GRUB_CMDLINE_LINUX_DEFAULT="apm=off" in /etc/default/grub. This has no effect. Does KVM do some power management on guests? If so, how do I turn this off for my LAMP guest? KVM doesn't do any power management of guests. But if everything is idle on the host (including your guest), then host power management could kick in. Have you tried playing with host pm? Could you try running your workload with the guest kernel parameter "idle=poll" and let me know the performance? Also, if you are running Linux 4.0 or later on the host, could you try running your workload with the KVM module parameter "halt_poll_ns=50"? Hansa-- Did you ever get a chance to run your workload with these changes? Hi David, Sorry I didn't respond to your mail earlier and nice of you to check back. To be honest no I haven't. To much work overload on my side. I've just added the idle=poll to the guests' grub and rebooted the server. I'll let it run a few day's to check if it helps. Btw: I'm running 2.6 linux kernel here so I'm not sure if I can add the "halt_poll_ns" param to KVM. Best, Hansa Wow! That makes real difference! The site was never that fast! Although CPU load is constantly on 100% any chance I can tweak this param? And is there a similar param for Windows guests? Since idle=poll gave a noticeable improvement, your workload is a good candidate for halt_poll_ns. This is a host KVM module parameter. It will give you a similar speedup, it will work independent of your guest os, and won't peg your cpu usage at 100% during periods of no activity. If you don't have halt_poll_ns available (your host kernel older than 4.0), is upgrading to a newer kernel or backporting the patch [1] an option? I'm not sure of any more guest params you can tweak (linux or windows), but I'll let you know if I come up with anything. [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926 Sorry David. That is not an option. It's a production server on 2.6 kernel. 4.0 kernel is not in Base repos and I don't know which impact installing the newer kernel version will have on this server. Any other options? -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
(adding back the kvm mailing list) On Tue, Jul 14, 2015 at 1:12 AM, "C. Bröcker" wrote: > On 14-7-2015 10:04, Hansa wrote: >> >> On 13-7-2015 20:57, David Matlack wrote: >>> >>> On Thu, Jun 18, 2015 at 10:26 AM, David Matlack >>> wrote: On Thu, Jun 18, 2015 at 1:25 AM, Hansa wrote: > > Hi, > > I have a LAMP server as guest in KVM. Whenever the server is idle for > some > time it takes about 30 seconds to load a Wordpress site. > If the server is not idle the site shows up in max 5 seconds. I've > already > turned of power management in the guest by passing > > GRUB_CMDLINE_LINUX_DEFAULT="apm=off" > > in /etc/default/grub. This has no effect. > Does KVM do some power management on guests? If so, how do I turn this > off > for my LAMP guest? KVM doesn't do any power management of guests. But if everything is idle on the host (including your guest), then host power management could kick in. Have you tried playing with host pm? Could you try running your workload with the guest kernel parameter "idle=poll" and let me know the performance? Also, if you are running Linux 4.0 or later on the host, could you try running your workload with the KVM module parameter "halt_poll_ns=50"? >>> >>> Hansa-- Did you ever get a chance to run your workload with these >>> changes? >> >> Hi David, >> >> Sorry I didn't respond to your mail earlier and nice of you to check back. >> To be honest no I haven't. To much work overload on my side. I've just added >> the idle=poll to the guests' grub and rebooted the server. I'll let it run a >> few day's to check if it helps. Btw: I'm running 2.6 linux kernel here so >> I'm not sure if I can add the "halt_poll_ns" param to KVM. >> >> Best, Hansa > > Wow! That makes real difference! > The site was never that fast! Although CPU load is constantly on 100% any > chance I can tweak this param? And is there a similar param for Windows > guests? Since idle=poll gave a noticeable improvement, your workload is a good candidate for halt_poll_ns. This is a host KVM module parameter. It will give you a similar speedup, it will work independent of your guest os, and won't peg your cpu usage at 100% during periods of no activity. If you don't have halt_poll_ns available (your host kernel older than 4.0), is upgrading to a newer kernel or backporting the patch [1] an option? I'm not sure of any more guest params you can tweak (linux or windows), but I'll let you know if I come up with anything. [1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=f7819512996361280b86259222456fcf15aad926 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: KVM slow LAMP guest
On Thu, Jun 18, 2015 at 1:25 AM, Hansa wrote: > Hi, > > I have a LAMP server as guest in KVM. Whenever the server is idle for some > time it takes about 30 seconds to load a Wordpress site. > If the server is not idle the site shows up in max 5 seconds. I've already > turned of power management in the guest by passing > > GRUB_CMDLINE_LINUX_DEFAULT="apm=off" > > in /etc/default/grub. This has no effect. > Does KVM do some power management on guests? If so, how do I turn this off > for my LAMP guest? KVM doesn't do any power management of guests. But if everything is idle on the host (including your guest), then host power management could kick in. Have you tried playing with host pm? Could you try running your workload with the guest kernel parameter "idle=poll" and let me know the performance? Also, if you are running Linux 4.0 or later on the host, could you try running your workload with the KVM module parameter "halt_poll_ns=50"? > > Best, Hansa > > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
KVM slow LAMP guest
Hi, I have a LAMP server as guest in KVM. Whenever the server is idle for some time it takes about 30 seconds to load a Wordpress site. If the server is not idle the site shows up in max 5 seconds. I've already turned of power management in the guest by passing GRUB_CMDLINE_LINUX_DEFAULT="apm=off" in /etc/default/grub. This has no effect. Does KVM do some power management on guests? If so, how do I turn this off for my LAMP guest? Best, Hansa -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html