== Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 
14:40:58 ==
---Problem Description---
I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy 
data" message being spammed to kern.log at about 100000 messages a second. 
Apart from original problem that triggers the error, the printk is a problem on 
its own. System might run out of space for logs, and would have high CPU load 
caused by journald.

This bug is to address an absence of rate limiter on printk at
https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268

Maybe wrapping the printk in printk_ratelimit() is all that is required?

if(printk_ratelimit()){
printk("KVM can't copy");
}

Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael
Folco / rfo...@br.ibm.com

---uname output---
4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le 
ppc64le ppc64le GNU/Linux

Machine Type = KVM guest on P8 8247 21L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. Deploy OpenStack with devstack
2. Run simultaneously between 4 and 6 instances of a script that
- boots a VM (openstack server create)
- immediately creates the backup of the VM (openstack server backup create)

This is not a very good way to reproduce it. I am still unable to
reproduce the problem just by using libvirt and qemu.

== Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07
09:42:31 ==

>From kernel logs
===========

[852719.001593] KVM can't copy data from 0x3fff86683c80!
[852719.001614] KVM can't copy data from 0x3fff86683c80!
[852719.001617] KVM can't copy data from 0x3fff66a83c80!
[852719.001624] KVM can't copy data from 0x3fff86683c80!
[852719.001634] KVM can't copy data from 0x3fff86683c80!
[852719.001637] KVM can't copy data from 0x3fff66a83c80!
[852719.001645] KVM can't copy data from 0x3fff86683c80!
[852719.001655] KVM can't copy data from 0x3fff86683c80!
[852719.001658] KVM can't copy data from 0x3fff66a83c80!
[852719.001665] KVM can't copy data from 0x3fff86683c80!
[852719.001668] KVM can't copy data from 0x3fff66a83c80!
[852719.001678] KVM can't copy data from 0x3fff66a83c80!
[852719.001686] KVM can't copy data from 0x3fff86683c80!

"KVM can't copy data from" error is being logged at very high rate
by kernel. rate limiting this error should fix this issue.

== Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 12:01:26 
==
Fix for this is submitted to PowerPC mailing list
and is available at below location:

https://patchwork.ozlabs.org/patch/728831/

It reads:

"KVM: PPC: Book3S: Ratelimit copy data failure error messages"

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Patch for the Kernel Team to evaluate.

Michael

On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> == Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 
> 14:40:58 ==
> ---Problem Description---
> I am experiencing a problem with KVM on Power with a symptom of "KVM can't 
> copy data" message being spammed to kern.log at about 100000 messages a 
> second. Apart from original problem that triggers the error, the printk is a 
> problem on its own. System might run out of space for logs, and would have 
> high CPU load caused by journald.
>
> This bug is to address an absence of rate limiter on printk at
> https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268
>
> Maybe wrapping the printk in printk_ratelimit() is all that is required?
>
>     if(printk_ratelimit()){
>        printk("KVM can't copy");
>     }
>
> Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / 
> rfo...@br.ibm.com
>
> ---uname output---
> 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le 
> ppc64le ppc64le GNU/Linux
>
> Machine Type = KVM guest on P8 8247 21L
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
>  1. Deploy OpenStack with devstack
> 2. Run simultaneously between 4 and 6 instances of a script that
>     - boots a VM (openstack server create)
>     - immediately creates the backup of the VM (openstack server backup 
> create)
>
> This is not a very good way to reproduce it. I am still unable to
> reproduce the problem just by using libvirt and qemu.
>
>
> == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 
> 09:42:31 ==
>
> >From kernel logs
> ===========
>
> [852719.001593] KVM can't copy data from 0x3fff86683c80!
> [852719.001614] KVM can't copy data from 0x3fff86683c80!
> [852719.001617] KVM can't copy data from 0x3fff66a83c80!
> [852719.001624] KVM can't copy data from 0x3fff86683c80!
> [852719.001634] KVM can't copy data from 0x3fff86683c80!
> [852719.001637] KVM can't copy data from 0x3fff66a83c80!
> [852719.001645] KVM can't copy data from 0x3fff86683c80!
> [852719.001655] KVM can't copy data from 0x3fff86683c80!
> [852719.001658] KVM can't copy data from 0x3fff66a83c80!
> [852719.001665] KVM can't copy data from 0x3fff86683c80!
> [852719.001668] KVM can't copy data from 0x3fff66a83c80!
> [852719.001678] KVM can't copy data from 0x3fff66a83c80!
> [852719.001686] KVM can't copy data from 0x3fff86683c80!
>
>
> "KVM can't copy data from" error is being logged at very high rate
> by kernel. rate limiting this error should fix this issue.
>
>
> == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 
> 12:01:26 ==
> Fix for this is submitted to PowerPC mailing list
> and is available at below location:
>
> https://patchwork.ozlabs.org/patch/728831/
>
> It reads:
>
> "KVM: PPC: Book3S: Ratelimit copy data failure error messages"
>
> ** Affects: ubuntu
>      Importance: Undecided
>      Assignee: Taco Screen team (taco-screen-team)
>          Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium 
> targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04 release.
== Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 
14:40:58 ==
---Problem Description---
I am experiencing a problem with KVM on Power with a symptom of "KVM can't copy 
data" message being spammed to kern.log at about 100000 messages a second. 
Apart from original problem that triggers the error, the printk is a problem on 
its own. System might run out of space for logs, and would have high CPU load 
caused by journald.

This bug is to address an absence of rate limiter on printk at
https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268

Maybe wrapping the printk in printk_ratelimit() is all that is required?

if(printk_ratelimit()){
printk("KVM can't copy");
}

Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael
Folco / rfo...@br.ibm.com

---uname output---
4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le 
ppc64le ppc64le GNU/Linux

Machine Type = KVM guest on P8 8247 21L

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. Deploy OpenStack with devstack
2. Run simultaneously between 4 and 6 instances of a script that
- boots a VM (openstack server create)
- immediately creates the backup of the VM (openstack server backup create)

This is not a very good way to reproduce it. I am still unable to
reproduce the problem just by using libvirt and qemu.

== Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07
09:42:31 ==

>From kernel logs
===========

[852719.001593] KVM can't copy data from 0x3fff86683c80!
[852719.001614] KVM can't copy data from 0x3fff86683c80!
[852719.001617] KVM can't copy data from 0x3fff66a83c80!
[852719.001624] KVM can't copy data from 0x3fff86683c80!
[852719.001634] KVM can't copy data from 0x3fff86683c80!
[852719.001637] KVM can't copy data from 0x3fff66a83c80!
[852719.001645] KVM can't copy data from 0x3fff86683c80!
[852719.001655] KVM can't copy data from 0x3fff86683c80!
[852719.001658] KVM can't copy data from 0x3fff66a83c80!
[852719.001665] KVM can't copy data from 0x3fff86683c80!
[852719.001668] KVM can't copy data from 0x3fff66a83c80!
[852719.001678] KVM can't copy data from 0x3fff66a83c80!
[852719.001686] KVM can't copy data from 0x3fff86683c80!

"KVM can't copy data from" error is being logged at very high rate
by kernel. rate limiting this error should fix this issue.

== Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 12:01:26 
==
Fix for this is submitted to PowerPC mailing list
and is available at below location:

https://patchwork.ozlabs.org/patch/728831/

It reads:

"KVM: PPC: Book3S: Ratelimit copy data failure error messages"

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Default Comment by Bridge

Patch for the Kernel Team to evaluate.

Michael

On 02/23/2017 10:11 AM, Launchpad Bug Tracker wrote:
> bugproxy (bugproxy) has assigned this bug to you for Ubuntu:
>
> == Comment: #0 - MIKHAIL S. MEDVEDEV <mmedv...@us.ibm.com> - 2017-02-03 
> 14:40:58 ==
> ---Problem Description---
> I am experiencing a problem with KVM on Power with a symptom of "KVM can't 
> copy data" message being spammed to kern.log at about 100000 messages a 
> second. Apart from original problem that triggers the error, the printk is a 
> problem on its own. System might run out of space for logs, and would have 
> high CPU load caused by journald.
>
> This bug is to address an absence of rate limiter on printk at
> https://github.com/torvalds/linux/blob/f64e8084c94bb0449177364856d8117e2f14c4c0/arch/powerpc/kvm/book3s_64_mmu.c#L268
>
> Maybe wrapping the printk in printk_ratelimit() is all that is required?
>
>     if(printk_ratelimit()){
>        printk("KVM can't copy");
>     }
>
> Contact Information = Mikhail Medvedev / mmedv...@us.ibm.com, Rafael Folco / 
> rfo...@br.ibm.com
>
> ---uname output---
> 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:53:20 UTC 2016 ppc64le 
> ppc64le ppc64le GNU/Linux
>
> Machine Type = KVM guest on P8 8247 21L
>
> ---Debugger---
> A debugger is not configured
>
> ---Steps to Reproduce---
>  1. Deploy OpenStack with devstack
> 2. Run simultaneously between 4 and 6 instances of a script that
>     - boots a VM (openstack server create)
>     - immediately creates the backup of the VM (openstack server backup 
> create)
>
> This is not a very good way to reproduce it. I am still unable to
> reproduce the problem just by using libvirt and qemu.
>
>
> == Comment: #5 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-07 
> 09:42:31 ==
>
> >From kernel logs
> ===========
>
> [852719.001593] KVM can't copy data from 0x3fff86683c80!
> [852719.001614] KVM can't copy data from 0x3fff86683c80!
> [852719.001617] KVM can't copy data from 0x3fff66a83c80!
> [852719.001624] KVM can't copy data from 0x3fff86683c80!
> [852719.001634] KVM can't copy data from 0x3fff86683c80!
> [852719.001637] KVM can't copy data from 0x3fff66a83c80!
> [852719.001645] KVM can't copy data from 0x3fff86683c80!
> [852719.001655] KVM can't copy data from 0x3fff86683c80!
> [852719.001658] KVM can't copy data from 0x3fff66a83c80!
> [852719.001665] KVM can't copy data from 0x3fff86683c80!
> [852719.001668] KVM can't copy data from 0x3fff66a83c80!
> [852719.001678] KVM can't copy data from 0x3fff66a83c80!
> [852719.001686] KVM can't copy data from 0x3fff86683c80!
>
>
> "KVM can't copy data from" error is being logged at very high rate
> by kernel. rate limiting this error should fix this issue.
>
>
> == Comment: #20 - VIPIN K. PARASHAR <vipar...@in.ibm.com> - 2017-02-23 
> 12:01:26 ==
> Fix for this is submitted to PowerPC mailing list
> and is available at below location:
>
> https://patchwork.ozlabs.org/patch/728831/
>
> It reads:
>
> "KVM: PPC: Book3S: Ratelimit copy data failure error messages"
>
> ** Affects: ubuntu
>      Importance: Undecided
>      Assignee: Taco Screen team (taco-screen-team)
>          Status: New
>
>
> ** Tags: architecture-ppc64le bugnameltc-151239 severity-medium 
> targetmilestone-inin1704

--
Michael Hohnbaum
OIL Program Manager
Power (ppc64el) Development Project Manager
Canonical, Ltd.

Please include this for Ubuntu 16.10, 16.04.02 along with Ubuntu 17.04
release.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1667416

Title:
  PowerNV: No rate limit for kernel error "KVM can't copy data from"

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667416/+subscriptions

-- 
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to