Re: [Intel-gfx] Kswapd 100% CPU since 3.8 on Sandybridge
On Mon, Oct 06, 2014 at 10:37:40AM +0100, Mel Gorman wrote: > On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote: > > Please excuse the non-wrapped email. My personal system is currently > > b0rked, so I'm sending this in frustration from my phone. > > > > My laptop is currently completely hosed. Disk light on full solid > > Mouse movement sluggish to the point of moving a couple cms per second. > > Firefox window greyed out but not OOM killed yet. When this behavior > > occurred in the past, if I ran top, I would see kswapd taking up 100% of > > one of my two CPUs. > > > > If I can catch the system in time before mouse movement becomes too > > sluggish, closing the browser window will cause kswapd usage to drop, and > > the system goes back to a normal state. If I don't catch it in time, I > > can't even ssh into the box to kill Firefox because the login times out. > > Occasionally Firefox gets OOM killed, but most of the time I have to use > > sysreq keys to reboot the system. > > > > This can be reproduced by using either Chrome or Firefox. Chrome fails > > faster. I'm not sure whether it's related to loading tabs with a bunch of > > images, maybe flash, but it takes around 10-15 tabs being open before it > > starts to fail. I can try to characterize it further. > > > > System: Lenovo x220 Intel Sandy Bridge graphics > > Ubuntu 14.04 with edgers PPA for Mesa > > 3.16.3 kernel > > > > Since around the 3.8 kernel time frame, I've been able to reproduce this > > behavior. I'm pretty sure it was a kernel change. > > > > I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a > > particular mm test. I still don't have time to triage this, but I'm now > > frustrated enough to make time. > > > > Mel, what test do you want me to run? > > > > Minimally I wanted you to sample the stack traces for kswapd, narrow down > to the time of its failure and see if it was stuck in a shrinker loop. What > I suspected at the time was that it was hammering on the i915 shrinker and > possibly doing repeated shrinks of the GPU objects in there. At one point > at least, that was an extremely heavy operation if the objections were > not freeable and I wanted to see if that was still the case. I confess I > haven't looked at the code to see what has changed recently. When kswapd is at 100% CPU usage, perf shows top symbols as: - 46.45% kswapd0 [i915] [k] i915_gem_inactive_count ◆ - i915_gem_inactive_count ▒ - 99.97% shrink_slab_node ▒ shrink_slab ▒ balance_pgdat ▒ kswapd ▒ kthread ▒ ret_from_fork ▒ - 15.83% kswapd0 [kernel.kallsyms][k] _raw_spin_lock ▒ - _raw_spin_lock ▒ + 47.36% list_lru_count_node ▒ + 25.31% grab_super_passive ▒ + 21.10% put_super ▒ + 5.19% super_cache_count ▒ + 0.96% mb_cache_shrink_count ▒ -5.22% kswapd0 [kernel.kallsyms][k] list_lru_count_node
Re: [Intel-gfx] Kswapd 100% CPU since 3.8 on Sandybridge
On Mon, Oct 6, 2014 at 11:37 AM, Mel Gorman wrote: > Minimally I wanted you to sample the stack traces for kswapd, narrow down > to the time of its failure and see if it was stuck in a shrinker loop. What > I suspected at the time was that it was hammering on the i915 shrinker and > possibly doing repeated shrinks of the GPU objects in there. At one point > at least, that was an extremely heavy operation if the objections were > not freeable and I wanted to see if that was still the case. I confess I > haven't looked at the code to see what has changed recently. We've stopped doing that with commit 2cfcd32a929b21c3cf77256dd8b858c076803ccc Author: Chris Wilson Date: Tue May 20 08:28:43 2014 +0100 drm/i915: Implement an oom-notifier for last resort shrinking so now we do the handbreak last ditch shrinking really only when the mm decided that it's time to oom. Until that point we should just do the proportional shrinking the vm asked us to, but not more. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [Intel-gfx] Kswapd 100% CPU since 3.8 on Sandybridge
On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote: > Please excuse the non-wrapped email. My personal system is currently > b0rked, so I'm sending this in frustration from my phone. > > My laptop is currently completely hosed. Disk light on full solid > Mouse movement sluggish to the point of moving a couple cms per second. > Firefox window greyed out but not OOM killed yet. When this behavior > occurred in the past, if I ran top, I would see kswapd taking up 100% of > one of my two CPUs. > > If I can catch the system in time before mouse movement becomes too > sluggish, closing the browser window will cause kswapd usage to drop, and > the system goes back to a normal state. If I don't catch it in time, I > can't even ssh into the box to kill Firefox because the login times out. > Occasionally Firefox gets OOM killed, but most of the time I have to use > sysreq keys to reboot the system. > > This can be reproduced by using either Chrome or Firefox. Chrome fails > faster. I'm not sure whether it's related to loading tabs with a bunch of > images, maybe flash, but it takes around 10-15 tabs being open before it > starts to fail. I can try to characterize it further. > > System: Lenovo x220 Intel Sandy Bridge graphics > Ubuntu 14.04 with edgers PPA for Mesa > 3.16.3 kernel > > Since around the 3.8 kernel time frame, I've been able to reproduce this > behavior. I'm pretty sure it was a kernel change. > > I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a > particular mm test. I still don't have time to triage this, but I'm now > frustrated enough to make time. > > Mel, what test do you want me to run? > Minimally I wanted you to sample the stack traces for kswapd, narrow down to the time of its failure and see if it was stuck in a shrinker loop. What I suspected at the time was that it was hammering on the i915 shrinker and possibly doing repeated shrinks of the GPU objects in there. At one point at least, that was an extremely heavy operation if the objections were not freeable and I wanted to see if that was still the case. I confess I haven't looked at the code to see what has changed recently. If that was confirmed then I to modify the mmtests Ftracereclaimcompact reported to focus exclusively on slab and give a breakdown of which shrinker it was spending time in. Right now, that reporter only says how much time is spent in slab which is not enough in this case. I just wanted to first know if it was worth the effort writing a monitor that gave a per-slab breakdown. If it can be both identified as shrinker-related and narrowed down to a specific shrinker then there is more to work with. mmtests can run in a monitor-only mode so it *should* be possible to turn on this monitor, wait for the problem to reproduce and focus on the end of the logs. Unfortunately, none of this explains why the machine completely froze. If it really was a shrinker problem then I expected the system to be extremely sluggish but did not predict that it would be so unresponsive that ssh was not an option. That has left me scratching my head. -- Mel Gorman SUSE Labs ___ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [Intel-gfx] Kswapd 100% CPU since 3.8 on Sandybridge
On Sat, Oct 04, 2014 at 10:05:20AM -0700, Sarah A Sharp wrote: > Please excuse the non-wrapped email. My personal system is currently > b0rked, so I'm sending this in frustration from my phone. > > My laptop is currently completely hosed. Disk light on full solid > Mouse movement sluggish to the point of moving a couple cms per second. > Firefox window greyed out but not OOM killed yet. When this behavior > occurred in the past, if I ran top, I would see kswapd taking up 100% of > one of my two CPUs. > > If I can catch the system in time before mouse movement becomes too > sluggish, closing the browser window will cause kswapd usage to drop, and > the system goes back to a normal state. If I don't catch it in time, I > can't even ssh into the box to kill Firefox because the login times out. > Occasionally Firefox gets OOM killed, but most of the time I have to use > sysreq keys to reboot the system. > > This can be reproduced by using either Chrome or Firefox. Chrome fails > faster. I'm not sure whether it's related to loading tabs with a bunch of > images, maybe flash, but it takes around 10-15 tabs being open before it > starts to fail. I can try to characterize it further. > > System: Lenovo x220 Intel Sandy Bridge graphics > Ubuntu 14.04 with edgers PPA for Mesa > 3.16.3 kernel > > Since around the 3.8 kernel time frame, I've been able to reproduce this > behavior. I'm pretty sure it was a kernel change. Hm, doesn't ring any bell for i915 bugs, but to make sure can you please sample debugfs/dri/0/i915_gem_objects while things go south? Thanks, Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch ___ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/intel-gfx
[Intel-gfx] Kswapd 100% CPU since 3.8 on Sandybridge
Please excuse the non-wrapped email. My personal system is currently b0rked, so I'm sending this in frustration from my phone. My laptop is currently completely hosed. Disk light on full solid Mouse movement sluggish to the point of moving a couple cms per second. Firefox window greyed out but not OOM killed yet. When this behavior occurred in the past, if I ran top, I would see kswapd taking up 100% of one of my two CPUs. If I can catch the system in time before mouse movement becomes too sluggish, closing the browser window will cause kswapd usage to drop, and the system goes back to a normal state. If I don't catch it in time, I can't even ssh into the box to kill Firefox because the login times out. Occasionally Firefox gets OOM killed, but most of the time I have to use sysreq keys to reboot the system. This can be reproduced by using either Chrome or Firefox. Chrome fails faster. I'm not sure whether it's related to loading tabs with a bunch of images, maybe flash, but it takes around 10-15 tabs being open before it starts to fail. I can try to characterize it further. System: Lenovo x220 Intel Sandy Bridge graphics Ubuntu 14.04 with edgers PPA for Mesa 3.16.3 kernel Since around the 3.8 kernel time frame, I've been able to reproduce this behavior. I'm pretty sure it was a kernel change. I mentioned this to Mel Gorman at LinuxCon NA, and he wanted me to run a particular mm test. I still don't have time to triage this, but I'm now frustrated enough to make time. Mel, what test do you want me to run? Sarah Sharp ___ Intel-gfx mailing list Intel-gfx@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/intel-gfx
