Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Doh! Now that I'm no longer using the 8 month old version of schedgraph.py that was displaying interesting but useless graphs, perhaps I won't continue to appear as though I'm raving like such a lunatic about what is contained in my ktr captures. Here follows a re-examination of the previously posted data with a recent schedgraph.py. Note that lacking any particular knowledge I'm only guessing at what might be significant. http://au.dyndns.ws/public/ktr-sched_gnome-netmonitor.out.bz2 (Stutter in glxgears with gnome metwork monitor) glxgears in runq for 236ms http://au.dyndns.ws/public/ktr-sched_move-glxgears_1.out.bz2 http://au.dyndns.ws/public/ktr-sched_move-glxgears_2.out.bz2 Nothing significant is apparent. http://au.dyndns.ws/public/ktr-sched_move-glxgears_3.out.bz2 (Dragging glxgears window which freezes - stops following mouse and stops updating, desktop doesn't respond to keyboard/mouse-clicks for the duration) glxgears in runq for 278ms and almost immediately again for 381ms. Note that this doesn't capture the entire period of interest which can be 1-2 seconds, due to the high number of context switches occurring with glxgears running and the difficulty of stopping the logging quickly after the event. I'll need a much larger (than 128k) buffer to catch an event in its entirety, unless someone can suggest a way to reduce the number of context switches significantly? http://au.dyndns.ws/public/ktr-sched-200801192346.out.bz2 Looks like this was probably just a mouse disconnect/reconnect - there is approx 1s of inactivity in irq20/ohci0 towards the end. Unfortunately these disconnects occur very frequently (typically multiple times per hour) since running with the KTR-enabled kernel (afaict). Unfortunately it looks as though that after gaining a false impression from this capture with the old schedgraph.py, I subsequently mis-interpreted numerous mouse disconnects as desktop freeze events. http://au.dyndns.ws/public/ktr-sched-200801200336.out.bz2 Looks like just another mouse disconnect. http://au.dyndns.ws/public/ktr-sched-200801200302.out.bz2 http://au.dyndns.ws/public/ktr-sched-200801210207.out.bz2 Nothing interesting is apparent. So it seems the only thing of interest that Ive managed to capture so far pertains to glxgears - an instance of the stutter and a part of a short freeze when dragging its window. Unfortunately these frequent mouse disconnects make it difficult to recognise genuine freezes during 'normal' use, if indeed they are still occurring with RELENG_7. However the glxgears behaviour remains (apparently) the same as it was on RELENG_6. Whether that's a telling sign or not remains to be seen. Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Wayne Sierke wrote: So it seems the only thing of interest that Ive managed to capture so far pertains to glxgears - an instance of the stutter and a part of a short freeze when dragging its window. Unfortunately these frequent mouse disconnects make it difficult to recognise genuine freezes during 'normal' use, if indeed they are still occurring with RELENG_7. However the glxgears behaviour remains (apparently) the same as it was on RELENG_6. Whether that's a telling sign or not remains to be seen. Wayne, thanks for continuing to investigate, since these little freezes definitely affect usability. If I can help in any way, let me know. I have not made any further graphs, but I continue to see intermittent mouse freezing (for short sub-seconf periods, usually). As for mouse disconnects, I don't know if that is what I am seeing, but one thing I do notice is that the keyboard is also affected (easily seen by holding down a key and letting it repeat - short pauses can be seen in the echo, which could be xterm, X, or the keyboard input, of course). Also, I tried unplugging my ps/2 mouse and using a USB one instead - same issue exists. In case this is scheduler-related, I tried running a CPU-hogging task (xtrs in model 4 mode, which spins, polling for input). While running this and moving the mouse rapidly between two windows (I use focus-under-mouse, so this causes focus events), I eventually get repeated short mouse freezes for quite some time (maybe 10 seconds) until things can catch up. This is not reproducible on Linux CFS (2.6.23) - the CPU use certainly affects event catching up in X, but the mouse stays smooth. Also, it seems that intermittent mouse freezes happen more often when I've been away from the machine for a while and return to start using the mouse again, but that's not always the case. A few short freezes/stutters happen a second or so after mouse movement resumes. -Joe ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
On Wed, 23 Jan 2008 08:27:58 -0700, Joe Peterson [EMAIL PROTECTED] wrote: Also, it seems that intermittent mouse freezes happen more often when I've been away from the machine for a while and return to start using the mouse again, but that's not always the case. A few short freezes/stutters happen a second or so after mouse movement resumes. -Joe Joe, I don't see any postings from you showing any ktr dumps. Do you have any? Your symptoms (that it seems to happen after you've been away for a while and then return and move the mouse) sound a lot like mine. I posted some ktr dumps and have since chatted off-list with Kris and Sam about what may be up. My dumps show the shared irq ath/pcm and the ath taskq are hogging the cpu for ages without the clock swi getting to run at all. Sam has suggested experimenting with the ath taskq priority and also with disabling ath bg scans which I will do, but right now I am back to looking at powerd again as the possible cause. I ran without powerd for a while when originally suggested by David Lawrence on Jan 12th. I believe I did still see freezes then, but I re-enabled powerd when I was ready to do LOCK_PROFILING and then ktr monitoring; I re-enabled it so I could be sure I had the same test conditions. At this point, I am no longer sure what happened when powerd was disabled. My recollection is that there were freezes while powerd was off, but the only email in which I appear to have posted about that says no freezes so far. So I'm running without powerd again, and at this point, several hours at the computer over two days, I have not seen further freezes. Does anyone else who sees these freezes also have powerd enabled and can try without powerd for a while? Since these freezes are proving so hard to pinpoint, it may be worth comparing notes to try to find things in common between the systems or eliminate other things. But first, it seems like we may be chasing three separate causes: 1. the softupdate freeze after removing a very large file (e.g., 1Gb) there is a noticeable freeze while the softupdate runs 2. the busy freeze folk complain of short freezes and mouse jerkiness while the system is busy, e.g., glxgears or compilations 3. the idle freeze short and longer freezes (some going into minutes) apparently when resuming work after having left the system mostly idle for a while Now, I also had the busy freeze when I first tested 7.0. At that time (several weeks back now) someone suggested switching to the ULE scheduler, which I did, and the symptoms I had were dramatically improved. Since then I've had occasions to run several compilations at once and had no mouse jerkiness. But for folk who still have it: what scheduler do you have and what processes are running when it happens? At the moment, I'm chasing an idle freeze. In other emails, I've posted details of what processes are typically running and several ktr dumps of such events. As noted, I'm looking again into powerd right now, and if that isn't it, I'll go to the ath taskq prio and scan stuff that Sam suggested. Anyone else with an idle freeze care to post details of what scheduler and processes are in use? -jr signature.asc Description: PGP signature
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
J.R. Oldroyd wrote: On Wed, 23 Jan 2008 08:27:58 -0700, Joe Peterson [EMAIL PROTECTED] wrote: Also, it seems that intermittent mouse freezes happen more often when I've been away from the machine for a while and return to start using the mouse again, but that's not always the case. A few short freezes/stutters happen a second or so after mouse movement resumes. -Joe Joe, I don't see any postings from you showing any ktr dumps. Do you have any? Your symptoms (that it seems to happen after you've been away for a while and then return and move the mouse) sound a lot like mine. Hi J.R., here is the post that contains links to my dumps: http://lists.freebsd.org/pipermail/freebsd-stable/2008-January/039599.html I posted some ktr dumps and have since chatted off-list with Kris and Sam about what may be up. My dumps show the shared irq ath/pcm and the ath taskq are hogging the cpu for ages without the clock swi getting to run at all. Sam has suggested experimenting with the ath taskq priority and also with disabling ath bg scans which I will do, but right now I am back to looking at powerd again as the possible cause. Hmm, well I don't have an Atheros on this machine - only ethernet. Also, I have not tried playing audio, so what I am seeing is simply with normal use. I ran without powerd for a while when originally suggested by David Lawrence on Jan 12th. I believe I did still see freezes then, but I re-enabled powerd when I was ready to do LOCK_PROFILING and then ktr monitoring; I re-enabled it so I could be sure I had the same test conditions. At this point, I am no longer sure what happened when powerd was disabled. My recollection is that there were freezes while powerd was off, but the only email in which I appear to have posted about that says no freezes so far. So I'm running without powerd again, and at this point, several hours at the computer over two days, I have not seen further freezes. Does anyone else who sees these freezes also have powerd enabled and can try without powerd for a while? Mine is a desktop machine, so I have not enabled powerd. Since these freezes are proving so hard to pinpoint, it may be worth comparing notes to try to find things in common between the systems or eliminate other things. But first, it seems like we may be chasing three separate causes: 1. the softupdate freeze after removing a very large file (e.g., 1Gb) there is a noticeable freeze while the softupdate runs 2. the busy freeze folk complain of short freezes and mouse jerkiness while the system is busy, e.g., glxgears or compilations 3. the idle freeze short and longer freezes (some going into minutes) apparently when resuming work after having left the system mostly idle for a while Now, I also had the busy freeze when I first tested 7.0. At that time (several weeks back now) someone suggested switching to the ULE scheduler, which I did, and the symptoms I had were dramatically improved. Since then I've had occasions to run several compilations at once and had no mouse jerkiness. But for folk who still have it: what scheduler do you have and what processes are running when it happens? I seem to see #2 (busy freezes). They are usually very short (sub-second) freezes, and they happen randomly as I move the mouse (well, I assume that I see it manifested in a mouse freeze, but it could very well be a system or X freeze, since I see it in keyboard key-held-down too). The mouse usually moves smoothly, but every once in a while, it sticks for a fraction of a second as I move it - irritating to say the least. Often the small freezes come in spurts, but they often are one at a time as well. When it comes in spurts, it is often shortly after moving the mouse after lots of idle time (as if the scheduler wakes up and has some fits for a short time - a non-scientific description ;). I am using ULE on 7.0. I'm also using ZFS (so the soft-updates issue doesn't apply, and I spoke with someone else who uses UFS2, not ZFS, and he said the mouse jerked around pretty badly in 7.0 on his machine). I started with using 4BSD under 7.0, of course, and yes, there were worse batches of freezes with it, especially when starting KDE and when compiling the kernel (it was nearly constant). With ULE, I no longer see compiles causing freezes, and generally the freezes are more subtle and shorter - in other words, ULE *is* better than 4BSD in this respect, but it is still worse than normal operation under FreeBSD 6.2 with 4BSD (although under 6.2/4BSD, I did see the occasional freeze during compiles - not nearly as obvious or often as 4BSD under 7.0). So, in my experience: 6.2/4BSD: smooth mouse motion all of the time except rare freezes during compiles. 7.0/4BSD: terrible mouse jerkiness under any load, especially when compiling. 7.0/ULE: smoother than 7.0/4BSD
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Kris, Latest captures (of interest) - they're not hard to come by, it seems. I see more than a couple of these every hour at least while I'm actively doing anything on this system, and I've witnessed up to two or three times that many at times. It seems my initial glee about catching Epiphany out was unfounded. Each capture has successively shown a different process/thread with the long exclusive cpu use, and the first one I've linked to here doesn't even show any active 'running' processes for the period in question, it has evolution and xorg 'yielding' and irq14 in 'runq'. The second one shows 'xorg' with exclusive 'running' for the duration. So it's proving quite variable, and after examining more than a handful of these I'm surmising that the 'freeze' is interfering with ktr logging somehow, resulting in the loss of one or more data points and resulting in the anomalous graphs as I mentioned in my previous message. I won't strive to take any more of these for now unless you indicate that you believe it could be useful. If I happen to catch any that look of particular interest I'll post them up and send links. In the meantime I'll revert to a standard GENERIC kernel, too, to gauge how much effect running with this KTR-enabled kernel on the frequency of these events, and the mouse disconnects. I'm keen to hear your take on what you think might be going on here. http://au.dyndns.ws/public/ktr-sched-200801200302.out.bz2 http://au.dyndns.ws/public/ktr-sched-200801210207.out.bz2 Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Kris, I caught this one during what I'd class as normal usage, with a handful of apps open, and no glxgears or other glx apps (intentionally) running. It shows Epiphany getting cpu solidly for 2 seconds! If you crank up the ticks per pixel and scroll to the end you can't miss it. This typifies an event that occurs regularly for me in my normal desktop use, albeit I normally use firefox with a large number of windows and tabs open. I'd long suspected that firefox might be involved in some of the pauses I've been experiencing. I'm quite chuffed about getting this one. http://au.dyndns.ws/public/ktr-sched-200801192346.out.bz2 Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Kris, Another one, this time xorg for 2 seconds, approximately in the middle. I'm feeling inclined to doubt the validity of this and the previous data set, however. Looking at the tail end of each of the events in question, there is clearly activity on other processes before the supposedly long exclusive cpu event terminates. Unless this is some limitation of the schedgraph.py script, or the ktr logging mechanism, etc., then this must surely indicate that they are not valid captures? I am all the more inclined to question them because the very next capture/dump I did after the previous set was captured looked so similar to it that I felt prompted to do a diff between the files and found that they contained identical lines for the most part, with only approx every nth line being different (for n=about 4 or so - I don't recall exactly and unfortunately didn't keep that particular one). Is it sufficient to just set debug.ktr.mask to resume capturing, or is it necessary to set debug.ktr.clear? (Note that so far there has always a substantial time interval between setting the mask and clearing it again, at least some minutes and usually much more, so I've assumed that there's plenty of logging occurring to over-write the previous contents). I should also have mentioned that I'm getting these messages periodically: ums0: at uhub0 port 2 (addr 2) disconnected ums0: detached ums0: B16_b_02 USB-PS/2 Optical Mouse, class 0/0, rev 2.00/98.02, addr 2 on uhub0 ums0: 8 buttons and Z dir. which for the moment I'm assuming are related to running the KTR-enabled kernel as I can't recall seeing these previously. http://au.dyndns.ws/public/ktr-sched-200801200336.out.bz2 Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
On Thu, 2008-01-17 at 20:50 +0100, Kris Kennaway wrote: Wayne Sierke wrote: On Mon, 2008-01-14 at 21:06 +0100, Kris Kennaway wrote: Same deal as before then. It cannot be the same problem as in the previous 6.x trace (unless you are using a non-mpsafe filesystem, i.e. not UFS). Kris In fact I have some MSDOSFS auto-mounting from /etc/fstab, but normally not in active use (by me - as for what gnome-* are doing in the background?). Without those mounts the stuttering in glxgears when 'Harddisk' is enabled in System Monitor is barely perceptible. The stuttering with Network Monitor remains as do the other freezes. I didn't have the same luck getting nvidia-driver working with LOCK_PROFILING in RELENG_7 as I did with MUTEX_PROFILING in RELENG_6. Looks like I'll have to test with nv or vesa driver instead, now. I've also prepared a KTR testing kernel but in the process realise I don't know which trace classes to include? KTR_SCHED Kris Ah, yes. Thanks. I've finally latched on to the whole schedgraph concept. Links to logs below. ktr-sched_gnome-netmonitor.out is just the 2Hz stutter in glxgears with the gnome Network Monitor applet active. I believe I've caught an instance of one stutter. ktr-sched_move-glxgears_1.out is a first attempt to catch the freeze that occurs when moving the glxgears window. Because of the freeze it's difficult to stop the recording promptly. It appears that all keyboard and mouse input (other than movement) is either ignored or discarded during the freeze[1]. Consequently, I have to wait until the system unfreezes before clicking over the terminal window to activate it and then pressing enter to run the waiting 'debug.ktr.mask=0' command. ktr-sched_move-glxgears_2.out is a second attempt. ktr-sched_move-glxgears_3.out was conducted by running the command: sleep 2; sysctl debug.ktr.mask=0 then waiting about 1.5 seconds, then dragging the glxgears window in an attempt to have the freeze active at the end of recording. What's a sensible upper limit for KTR_ENTRIES? Wayne [1] With the very curious exception that when the freeze occurs due to an alt-tab window-switching action, mouse and keyboard *do* get buffered - if I continue hitting alt-tab after the system freezes, those alt-tabs get played out when the system unfreezes. Similarly if I click over a window while the system is frozen during alt-tab, that mouse click gets played as normal when the system unfreezes. Odd. http://au.dyndns.ws/public/ktr-sched_gnome-netmonitor.out.bz2 http://au.dyndns.ws/public/ktr-sched_move-glxgears_1.out.bz2 http://au.dyndns.ws/public/ktr-sched_move-glxgears_2.out.bz2 http://au.dyndns.ws/public/ktr-sched_move-glxgears_3.out.bz2 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
On Mon, 2008-01-14 at 21:06 +0100, Kris Kennaway wrote: Same deal as before then. It cannot be the same problem as in the previous 6.x trace (unless you are using a non-mpsafe filesystem, i.e. not UFS). Kris In fact I have some MSDOSFS auto-mounting from /etc/fstab, but normally not in active use (by me - as for what gnome-* are doing in the background?). Without those mounts the stuttering in glxgears when 'Harddisk' is enabled in System Monitor is barely perceptible. The stuttering with Network Monitor remains as do the other freezes. I didn't have the same luck getting nvidia-driver working with LOCK_PROFILING in RELENG_7 as I did with MUTEX_PROFILING in RELENG_6. Looks like I'll have to test with nv or vesa driver instead, now. I've also prepared a KTR testing kernel but in the process realise I don't know which trace classes to include? Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Wayne Sierke wrote: On Mon, 2008-01-14 at 21:06 +0100, Kris Kennaway wrote: Same deal as before then. It cannot be the same problem as in the previous 6.x trace (unless you are using a non-mpsafe filesystem, i.e. not UFS). Kris In fact I have some MSDOSFS auto-mounting from /etc/fstab, but normally not in active use (by me - as for what gnome-* are doing in the background?). Without those mounts the stuttering in glxgears when 'Harddisk' is enabled in System Monitor is barely perceptible. The stuttering with Network Monitor remains as do the other freezes. I didn't have the same luck getting nvidia-driver working with LOCK_PROFILING in RELENG_7 as I did with MUTEX_PROFILING in RELENG_6. Looks like I'll have to test with nv or vesa driver instead, now. I've also prepared a KTR testing kernel but in the process realise I don't know which trace classes to include? KTR_SCHED Kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Kris Kennaway wrote: KTR_SCHED Kris, BTW, I am curious if the traces I posted were informative. Let me know if I did not create them correctly. The xterm test seems to vary in usefulness depending on video card (faster cards catch up too quickly), but the freezing still happens quite often using apps like firefox, especially. Here's the post link: http://lists.freebsd.org/pipermail/freebsd-stable/2008-January/039599.html Thanks, Joe ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
On Sun, 2008-01-13 at 21:09 +0100, Kris Kennaway wrote: Yeah, this shows things like contention between the mouse device and other parts of the kernel that still require the Giant lock in 6.x. It is not likely that these will be fixed in 6.x but most of them are in 7.0, so you should obtain better performance by upgrading to 7.0. Kris This system is now: FreeBSD 7.0-PRERELEASE #0: Mon Jan 14 17:40:02 CST 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/GENERIC i386 I haven't touched ports yet apart from nvidia-driver- and adding compat6x. I haven't done any extensive testing as yet but repeating what I was doing previously as a 'litmus test' - launch gnome, open a terminal, launch glxgears, (1) drag glxgears window, (2) add gnome 'Network Monitor' applet, (3) add 'System Monitor' applet with 'Harddisk' enabled as a 'Monitored Resource'. So far I'm seeing what seems to be identical behaviour: (1) Huge lag when I commence dragging glxgears window, typ. 1-2 seconds (2) glxgears stutters at 2Hz matching update rate of applet (3) ditto (1) manifests in two ways - window drags but glxgears freezes, and, more severely, window freezes where it is (stops both updating and following cursor). So far while launching/using evolution, epiphany, etc. I see stalls that look pretty similar to 6.x behaviour. Just thought I'd check in in case there are any suggestions at this point. If not I'll proceed with recompiling ports and preparing test kernels. Wayne ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: 7.0-PRERELEASE desktop system periodically freezes momentarily
Wayne Sierke wrote: Just thought I'd check in in case there are any suggestions at this point. If not I'll proceed with recompiling ports and preparing test kernels. Same deal as before then. It cannot be the same problem as in the previous 6.x trace (unless you are using a non-mpsafe filesystem, i.e. not UFS). Kris ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
