== Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 14:40:58 == ---Problem Description--- I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald.
This bug is to address an absence of rate limiter on printk at https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268 Maybe wrapping the printk in printk_ratelimit() is all that is required? if(printk_ratelimit()){ printk("KVM can't copy"); } Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / rfo...@br.ibm.com ---uname output--- 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = KVM guest on P8 8247 21L ---Debugger--- A debugger is not configured ---Steps to Reproduce--- 1. Deploy OpenStack with devstack 2. Run simultaneously between 4 and 6 instances of a script that - boots a VM (openstack server create) - immediately creates the backup of the VM (openstack server backup create) This is not a very good way to reproduce it. I am still unable to reproduce the problem just by using libvirt and qemu. == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 09:42:31 == >From kernel logs =========== [852719.001593] KVM can't copy data from 0x3fff86683c80! [852719.001614] KVM can't copy data from 0x3fff86683c80! [852719.001617] KVM can't copy data from 0x3fff66a83c80! [852719.001624] KVM can't copy data from 0x3fff86683c80! [852719.001634] KVM can't copy data from 0x3fff86683c80! [852719.001637] KVM can't copy data from 0x3fff66a83c80! [852719.001645] KVM can't copy data from 0x3fff86683c80! [852719.001655] KVM can't copy data from 0x3fff86683c80! [852719.001658] KVM can't copy data from 0x3fff66a83c80! [852719.001665] KVM can't copy data from 0x3fff86683c80! [852719.001668] KVM can't copy data from 0x3fff66a83c80! [852719.001678] KVM can't copy data from 0x3fff66a83c80! [852719.001686] KVM can't copy data from 0x3fff86683c80! "KVM can't copy data from" error is being logged at very high rate by kernel. rate limiting this error should fix this issue. == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 12:01:26 == Fix for this is submitted to PowerPC mailing list and is available at below location: https://patchwork.ozlabs.org/patch/728831/ It reads: "KVM: PPC: Book3S: Ratelimit copy data failure error messages" Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Patch for the Kernel Team to evaluate. Michael On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote: > bugproxy (bugproxy) has assigned this bug to you for Ubuntu: > > == Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 > 14:40:58 == > ---Problem Description--- > I am experiencing a problem with KVM on Power with a symptom of "KVM can't > copy data" message being spammed to kern.log at about 100000 messages a > second. Apart from original problem that triggers the error, the printk is a > problem on its own. System might run out of space for logs, and would have > high CPU load caused by journald. > > This bug is to address an absence of rate limiter on printk at > https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268 > > Maybe wrapping the printk in printk_ratelimit() is all that is required? > > if(printk_ratelimit()){ > printk("KVM can't copy"); > } > > Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / > rfo...@br.ibm.com > > ---uname output--- > 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le > ppc64le ppc64le GNU/Linux > > Machine Type = KVM guest on P8 8247 21L > > ---Debugger--- > A debugger is not configured > > ---Steps to Reproduce--- > 1. Deploy OpenStack with devstack > 2. Run simultaneously between 4 and 6 instances of a script that > - boots a VM (openstack server create) > - immediately creates the backup of the VM (openstack server backup > create) > > This is not a very good way to reproduce it. I am still unable to > reproduce the problem just by using libvirt and qemu. > > > == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 > 09:42:31 == > > >From kernel logs > =========== > > [852719.001593] KVM can't copy data from 0x3fff86683c80! > [852719.001614] KVM can't copy data from 0x3fff86683c80! > [852719.001617] KVM can't copy data from 0x3fff66a83c80! > [852719.001624] KVM can't copy data from 0x3fff86683c80! > [852719.001634] KVM can't copy data from 0x3fff86683c80! > [852719.001637] KVM can't copy data from 0x3fff66a83c80! > [852719.001645] KVM can't copy data from 0x3fff86683c80! > [852719.001655] KVM can't copy data from 0x3fff86683c80! > [852719.001658] KVM can't copy data from 0x3fff66a83c80! > [852719.001665] KVM can't copy data from 0x3fff86683c80! > [852719.001668] KVM can't copy data from 0x3fff66a83c80! > [852719.001678] KVM can't copy data from 0x3fff66a83c80! > [852719.001686] KVM can't copy data from 0x3fff86683c80! > > > "KVM can't copy data from" error is being logged at very high rate > by kernel. rate limiting this error should fix this issue. > > > == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 > 12:01:26 == > Fix for this is submitted to PowerPC mailing list > and is available at below location: > > https://patchwork.ozlabs.org/patch/728831/ > > It reads: > > "KVM: PPC: Book3S: Ratelimit copy data failure error messages" > > ** Affects: ubuntu > Importance: Undecided > Assignee: Taco Screen team (taco-screen-team) > Status: New > > > ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium > targetmilestone-inin1704 -- Michael Hohnbaum OIL Program Manager Power (ppc64el) Development Project Manager Canonical, Ltd. Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04 release. == Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 14:40:58 == ---Problem Description--- I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy data" message being spammed to kern.log at about 100000 messages a second. Apart from original problem that triggers the error, the printk is a problem on its own. System might run out of space for logs, and would have high CPU load caused by journald. This bug is to address an absence of rate limiter on printk at https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268 Maybe wrapping the printk in printk_ratelimit() is all that is required? if(printk_ratelimit()){ printk("KVM can't copy"); } Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / rfo...@br.ibm.com ---uname output--- 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux Machine Type = KVM guest on P8 8247 21L ---Debugger--- A debugger is not configured ---Steps to Reproduce--- 1. Deploy OpenStack with devstack 2. Run simultaneously between 4 and 6 instances of a script that - boots a VM (openstack server create) - immediately creates the backup of the VM (openstack server backup create) This is not a very good way to reproduce it. I am still unable to reproduce the problem just by using libvirt and qemu. == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 09:42:31 == >From kernel logs =========== [852719.001593] KVM can't copy data from 0x3fff86683c80! [852719.001614] KVM can't copy data from 0x3fff86683c80! [852719.001617] KVM can't copy data from 0x3fff66a83c80! [852719.001624] KVM can't copy data from 0x3fff86683c80! [852719.001634] KVM can't copy data from 0x3fff86683c80! [852719.001637] KVM can't copy data from 0x3fff66a83c80! [852719.001645] KVM can't copy data from 0x3fff86683c80! [852719.001655] KVM can't copy data from 0x3fff86683c80! [852719.001658] KVM can't copy data from 0x3fff66a83c80! [852719.001665] KVM can't copy data from 0x3fff86683c80! [852719.001668] KVM can't copy data from 0x3fff66a83c80! [852719.001678] KVM can't copy data from 0x3fff66a83c80! [852719.001686] KVM can't copy data from 0x3fff86683c80! "KVM can't copy data from" error is being logged at very high rate by kernel. rate limiting this error should fix this issue. == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 12:01:26 == Fix for this is submitted to PowerPC mailing list and is available at below location: https://patchwork.ozlabs.org/patch/728831/ It reads: "KVM: PPC: Book3S: Ratelimit copy data failure error messages" Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Default Comment by Bridge Patch for the Kernel Team to evaluate. Michael On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote: > bugproxy (bugproxy) has assigned this bug to you for Ubuntu: > > == Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 > 14:40:58 == > ---Problem Description--- > I am experiencing a problem with KVM on Power with a symptom of "KVM can't > copy data" message being spammed to kern.log at about 100000 messages a > second. Apart from original problem that triggers the error, the printk is a > problem on its own. System might run out of space for logs, and would have > high CPU load caused by journald. > > This bug is to address an absence of rate limiter on printk at > https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268 > > Maybe wrapping the printk in printk_ratelimit() is all that is required? > > if(printk_ratelimit()){ > printk("KVM can't copy"); > } > > Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / > rfo...@br.ibm.com > > ---uname output--- > 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le > ppc64le ppc64le GNU/Linux > > Machine Type = KVM guest on P8 8247 21L > > ---Debugger--- > A debugger is not configured > > ---Steps to Reproduce--- > 1. Deploy OpenStack with devstack > 2. Run simultaneously between 4 and 6 instances of a script that > - boots a VM (openstack server create) > - immediately creates the backup of the VM (openstack server backup > create) > > This is not a very good way to reproduce it. I am still unable to > reproduce the problem just by using libvirt and qemu. > > > == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 > 09:42:31 == > > >From kernel logs > =========== > > [852719.001593] KVM can't copy data from 0x3fff86683c80! > [852719.001614] KVM can't copy data from 0x3fff86683c80! > [852719.001617] KVM can't copy data from 0x3fff66a83c80! > [852719.001624] KVM can't copy data from 0x3fff86683c80! > [852719.001634] KVM can't copy data from 0x3fff86683c80! > [852719.001637] KVM can't copy data from 0x3fff66a83c80! > [852719.001645] KVM can't copy data from 0x3fff86683c80! > [852719.001655] KVM can't copy data from 0x3fff86683c80! > [852719.001658] KVM can't copy data from 0x3fff66a83c80! > [852719.001665] KVM can't copy data from 0x3fff86683c80! > [852719.001668] KVM can't copy data from 0x3fff66a83c80! > [852719.001678] KVM can't copy data from 0x3fff66a83c80! > [852719.001686] KVM can't copy data from 0x3fff86683c80! > > > "KVM can't copy data from" error is being logged at very high rate > by kernel. rate limiting this error should fix this issue. > > > == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 > 12:01:26 == > Fix for this is submitted to PowerPC mailing list > and is available at below location: > > https://patchwork.ozlabs.org/patch/728831/ > > It reads: > > "KVM: PPC: Book3S: Ratelimit copy data failure error messages" > > ** Affects: ubuntu > Importance: Undecided > Assignee: Taco Screen team (taco-screen-team) > Status: New > > > ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium > targetmilestone-inin1704 -- Michael Hohnbaum OIL Program Manager Power (ppc64el) Development Project Manager Canonical, Ltd. Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04 release. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1667416 Title: PowerNV: No rate limit for kernel error "KVM can't copy data from" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667416/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs