Some pulseaudio questions...
Folks, Anyone want to clarify my understanding of PA's use of mlock/posix_madvise? From looking over the code - in particular pa_will_need, and its callsites - it looks like PA doesn't really use this support that it has for locking pages. Which seems weird. I'll admit I'm about ready to hack in some horrible mlockall hacks to see if that'll stop the poppy/skippyness on this box. Jon. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, 05.06.09 00:21, Jon Masters (jonat...@jonmasters.org) wrote: > Folks, > > Anyone want to clarify my understanding of PA's use of > mlock/posix_madvise? From looking over the code - in particular > pa_will_need, and its callsites - it looks like PA doesn't really use > this support that it has for locking pages. Which seems weird. mlock() is not actually used anymore by PA on modern kernels. The longer story goes like this: Text book RT applications use mlock()/mlockall() to lock themselves into memory and make sure they never are swapped out. This is something we cannot really do for PA given that map a *lot* of stuff into our address space: libraries, SHM segments for communications with other clients, cached samples, and so on. If we'd lock all that into memory there wouldn't be any memory left for much else, i.e. all other programs would have to share what is remaining and would have to swap all the time. Given that PA is just an auxiliary tool and not the main reason people run computers this is hence not an option. Due to that PA doesn't use mlock()/mlockall() like textbooks suggest, and it never did. Now, just ignoring the whole locking issue is also not an option. So I tried to find a compromise: try my best to make sure that the data accessed by the RT threads is available in memory but ignore the rest of the memory. To achieve that I needed a way to safely swap in memory when *I* need it to instead of letting the kernel to do as it as late as possible. Then, whenever the non-RT threads in PA pass off data to the RT threads I first swap the data in explicitly. That way I hope that the RT threads never need to wait for swapping in and can continue to loop in their little loops and continue to do whatever else they might want to do, but not wait for disks spinning. There is a system call for requesting swapping in of memory: posix_madvise(POSIX_MADV_WILLNEED). A few kernel releases ago this didn't work for anonymous memory, only for file-backed memory. (But on my request this was then changed in the kernel). Now, to support older kernels I added a dirty dirty hack: I try to lock those pages into memory and then unlock them right away, which should have about the same effect as the madvise() call. On current kernels that mlock() is never tried however, because WILLNEEED works fine anyway. And it's a horrible hack. And I probably should remove this from PA now. > I'll admit I'm about ready to hack in some horrible mlockall hacks to > see if that'll stop the poppy/skippyness on this box. I doubt that locking PA into memory will fix your issues. If PA drops out often this might have various reasons, but probably not swapping. Often the timing calls of your sound driver are inaccurate and cause PA to miss its deadlines. To work around that you could try disabling timer-based scheduling mode and revert to sound card IRQ scheduled playback. For that try passing "tsched=0" to module-hal-detect in /etc/pulse/default.pa. Then restart PA. Another issue might be that PA does not actually get scheduled often enough by the kernel. Might be caused by a bad driver (nvidia closed source). You can run PA as RT if you wish (which we hopfully will be able to enable by default in F12). For that increase RLIMIT_RTPRIO to 10 or so in /etc/security/limits.conf and login again. Usually running "pulseaudio -v" in a terminal might give you a hint what might be going wrong. Lennart -- Lennart PoetteringRed Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On 06/05/2009 06:57 AM, Lennart Poettering wrote: > Usually running "pulseaudio -v" in a terminal might give you a > hint what might be going wrong. Lennart, Maybe this is a stupid question (you know I am constantly full of them), but is there any way for pulseaudio to detect this "common" condition where the audio is skipping and inform the user of the possible workarounds (maybe a DBUS popup or something directly to syslog). I'm just considering that the average user might not make the leap to running "pulseaudio -v" in a terminal to figuring this out. Just brainstorming, ~spot -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, 05.06.09 09:01, Tom spot Callaway (tcall...@redhat.com) wrote: > > On 06/05/2009 06:57 AM, Lennart Poettering wrote: > > Usually running "pulseaudio -v" in a terminal might give you a > > hint what might be going wrong. > > Lennart, > > Maybe this is a stupid question (you know I am constantly full of them), > but is there any way for pulseaudio to detect this "common" condition > where the audio is skipping and inform the user of the possible > workarounds (maybe a DBUS popup or something directly to syslog). I'm > just considering that the average user might not make the leap to > running "pulseaudio -v" in a terminal to figuring this out. We are already doing this. We verify all timing related data we get from the driver as far as possible. i.e. everything returned by snd_pcm_delay() and snd_pcm_avail() is checked whether it is in specific bounds. If it is not we log that to syslog and clamp it to something which makes more sense to us. But of couse, there's only so much you can catch and even less than you can fix by clamping with that. In fact this turned out to be very useful to track down a couple of timing overflows in the HDA driver that have now been fixed in the F11 kernel. However there are still some issues left. Just search for "snd_pcm_delay"/"snd_pcm_avail"/"snd_pcm_update_avail" in bugzilla. Lennart -- Lennart PoetteringRed Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, 2009-06-05 at 12:57 +0200, Lennart Poettering wrote: > On Fri, 05.06.09 00:21, Jon Masters (jonat...@jonmasters.org) wrote: > > > Folks, > > > > Anyone want to clarify my understanding of PA's use of > > mlock/posix_madvise? From looking over the code - in particular > > pa_will_need, and its callsites - it looks like PA doesn't really use > > this support that it has for locking pages. Which seems weird. > > mlock() is not actually used anymore by PA on modern kernels. I noticed :) But then, neither is the posix_madvise being used that much either, as I noted. On a related note, I have revived the upstream discussion of MCL_INHERIT and will repost patches implementing this over the weekend - then I can have a simple wrapper utility to test forcing an app like PA to do an mlockall. > Text book RT applications use mlock()/mlockall() to lock themselves > into memory and make sure they never are swapped out. This is > something we cannot really do for PA given that map a *lot* of stuff > into our address space: libraries, SHM segments for communications > with other clients, cached samples, and so on. If we'd lock all that > into memory there wouldn't be any memory left for much else Yeah, I'm aware of this. But there perhaps should be some option anyway - after all, you already have all the support code for it, and already handle setting real time priorities too. In my brief time with a hacked up local build that does an mlockall right at the beginning of the mainloop, I am hearing few audio pops and skips on this box. It's obviously not a longer term solution, just a datapoint. > There is a system call for requesting swapping in of memory: > posix_madvise(POSIX_MADV_WILLNEED). Yeah, I saw that and I think it's a nice idea. I'm not sure it's being called very often though - bear in mind I only spent a few minutes looking at the source, so I might have missed something. > > I'll admit I'm about ready to hack in some horrible mlockall hacks to > > see if that'll stop the poppy/skippyness on this box. It did drastically reduce it. The box is under extremely heavy stress as it's running a lot of stuff - dozens of firefox, many evolution windows, IRC, rhythmbox, a few VMs, etc. I'm looking forward to the IO Controller patches so I can e.g. prevent evolution from hogging more than a certain amount of disk bandwidth, which I'm sure will help. > I doubt that locking PA into memory will fix your issues. If PA drops > out often this might have various reasons, but probably not > swapping. Often the timing calls of your sound driver are inaccurate > and cause PA to miss its deadlines. The Intel HDA driver on here doesn't appear to be in your badlist any more, but maybe I am wrong: 00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High Definition Audio Controller (rev 02) > To work around that you could try > disabling timer-based scheduling mode and revert to sound card IRQ > scheduled playback. For that try passing "tsched=0" to > module-hal-detect in /etc/pulse/default.pa. I will try that. > Then restart PA. Another issue might be that PA does not actually > get scheduled often enough by the kernel. I don't think it's that. There's no closed source driver installed. I also am currently running PA as an RT task and my SMI detector shows that this box is not disappearing off into SMI land too often. > Usually running "pulseaudio -v" in a terminal might give you a > hint what might be going wrong. Not very much useful output when I set it into a debug loglevel - maybe this is different. Jon. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, 2009-06-05 at 11:11 -0400, Jon Masters wrote: > On Fri, 2009-06-05 at 12:57 +0200, Lennart Poettering wrote: > > On Fri, 05.06.09 00:21, Jon Masters (jonat...@jonmasters.org) wrote: > > Text book RT applications use mlock()/mlockall() to lock themselves > > into memory and make sure they never are swapped out. This is > > something we cannot really do for PA given that map a *lot* of stuff > > into our address space: libraries, SHM segments for communications > > with other clients, cached samples, and so on. If we'd lock all that > > into memory there wouldn't be any memory left for much else > > Yeah, I'm aware of this. But there perhaps should be some option anyway > - after all, you already have all the support code for it, and already > handle setting real time priorities too. In my brief time with a hacked > up local build that does an mlockall right at the beginning of the > mainloop, I am hearing few audio pops and skips on this box. It's > obviously not a longer term solution, just a datapoint. I'll join the PA devel list over the weekend, it's not strictly that Fedora specific now. But one thing I'm wondering is whether you might benefit from splitting PA into a small core-util bit that were lockable and having all the rest outside in separate tasks - that's probably not too feasible at this stage though. I want to help fix whatever problems I'm getting on each of my machines running PA, rather than sound like I'm trashing talking PA as a technology. The sad reality is that Linux audio worked for me more smoothly 14 years ago when I started with Linux (and manually had to set jumpers, run isapnpdump, etc.) than it does now. It was smoother when I had early ESD[0] than it is now, and smoother when I first built experimental ALSA drivers than it is now. Things like PA are a great concept in theory, but they're not of much benefit if (as in my case) the only obvious way I can get an decent experience is to hack my system and run stuff under pasuspender. Jon. [0] And I mean alongside 0.1 enlightment, back when I had the Free Software Song as a ringtone and enjoyed hearing the startup pips as ESD opened the sound device. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, Jun 5, 2009 at 5:11 PM, Jon Masters wrote: > The Intel HDA driver on here doesn't appear to be in your badlist any > more, but maybe I am wrong: > > 00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High > Definition Audio Controller (rev 02) Because HDA is too generic, it depends on the codec that is used by your soundcard. I have two HDA devices here one with ALC881 and one with AD1988, the later works a lot better with PA. (even thought with the recent fixes in F11 the ALC is working much better as in the past). -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Fri, 05.06.09 11:11, Jon Masters (jonat...@jonmasters.org) wrote: > > > Anyone want to clarify my understanding of PA's use of > > > mlock/posix_madvise? From looking over the code - in particular > > > pa_will_need, and its callsites - it looks like PA doesn't really use > > > this support that it has for locking pages. Which seems weird. > > > > mlock() is not actually used anymore by PA on modern kernels. > > I noticed :) But then, neither is the posix_madvise being used that much > either, as I noted. Yes, that is true. We primarily use that when playing samples from the cache, since the sample cache has a very high and obvious probability to be swapped out. We probably should place these calls at other strategic points. > > Text book RT applications use mlock()/mlockall() to lock themselves > > into memory and make sure they never are swapped out. This is > > something we cannot really do for PA given that map a *lot* of stuff > > into our address space: libraries, SHM segments for communications > > with other clients, cached samples, and so on. If we'd lock all that > > into memory there wouldn't be any memory left for much else > > Yeah, I'm aware of this. But there perhaps should be some option anyway > - after all, you already have all the support code for it, and already > handle setting real time priorities too. In my brief time with a hacked > up local build that does an mlockall right at the beginning of the > mainloop, I am hearing few audio pops and skips on this box. It's > obviously not a longer term solution, just a datapoint. Hmm, interesting. You really say mlockall() has a big effect? An option to use mlockall certainly does make sense. I have added that now to PA git in commit a9b38b35. > It did drastically reduce it. The box is under extremely heavy stress as > it's running a lot of stuff - dozens of firefox, many evolution windows, > IRC, rhythmbox, a few VMs, etc. I'm looking forward to the IO Controller > patches so I can e.g. prevent evolution from hogging more than a certain > amount of disk bandwidth, which I'm sure will help. What would be good to have would be a "swap niceness" value that could be attached to a processes or memory regions. i.e. some way to influence the swapping algorithm in a less binary way than just "swap this" or "don't swap this". > > I doubt that locking PA into memory will fix your issues. If PA drops > > out often this might have various reasons, but probably not > > swapping. Often the timing calls of your sound driver are inaccurate > > and cause PA to miss its deadlines. > > The Intel HDA driver on here doesn't appear to be in your badlist any > more, but maybe I am wrong: > > 00:1b.0 Audio device: Intel Corporation 82801G (ICH7 Family) High > Definition Audio Controller (rev 02) The actualy codec used is more interesting. The codec is shown at the top of the 'alsamixer' screen. Lennart -- Lennart PoetteringRed Hat, Inc. lennart [at] poettering [dot] net http://0pointer.net/lennart/ GnuPG 0x1A015CC4 -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Sun, Jun 07, 2009 at 12:51:02AM +0200, Lennart Poettering wrote: > What would be good to have would be a "swap niceness" value that could > be attached to a processes or memory regions. i.e. some way to > influence the swapping algorithm in a less binary way than just "swap > this" or "don't swap this". > I've been working on this along with some enhancements to make fadvise more than utterly useless... hopefully will be ready for posting soon. cheers, Kyle -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Sat, 2009-06-06 at 19:01 -0400, Kyle McMartin wrote: > On Sun, Jun 07, 2009 at 12:51:02AM +0200, Lennart Poettering wrote: > > What would be good to have would be a "swap niceness" value that could > > be attached to a processes or memory regions. i.e. some way to > > influence the swapping algorithm in a less binary way than just "swap > > this" or "don't swap this". > I've been working on this along with some enhancements to make fadvise > more than utterly useless... hopefully will be ready for posting soon. Main problem is proving its utility with actual benchmarks - witness the recent first class citizen VM patch thread. Jon. -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
Re: Some pulseaudio questions...
On Mon, Jun 08, 2009 at 03:24:00AM -0400, Jon Masters wrote: > On Sat, 2009-06-06 at 19:01 -0400, Kyle McMartin wrote: > > On Sun, Jun 07, 2009 at 12:51:02AM +0200, Lennart Poettering wrote: > > > What would be good to have would be a "swap niceness" value that could > > > be attached to a processes or memory regions. i.e. some way to > > > influence the swapping algorithm in a less binary way than just "swap > > > this" or "don't swap this". > > > I've been working on this along with some enhancements to make fadvise > > more than utterly useless... hopefully will be ready for posting soon. > > Main problem is proving its utility with actual benchmarks - witness the > recent first class citizen VM patch thread. > Nrgh, hate reply-to headers. Anyway, this is not one of those 'subjective' things like tweaking schedulers or pageout algorithms. It's about giving developers the ability to control pageout and replacement policy. For instance, if you're working on gvfs, and are copying a file to a remote host, what are the odds that you want to waste 4GB of memory caching the entireity of it? Almost zero. If it's a video, you may want to keep the first, say, 5%, and then recycle another 5% as a window while you copy. (The first 5% in case the user decides to play it.) If you're running a large webserver, say, mirrors.kernel.org, you may want your most-commonly-accessed files to stay in page cache. You wouldn't want someone downloading a whole bunch of ancient kernels to cause 2.6.30 to get evicted... Providing this control is important, because it's impossible to do one-size-fits-all without hints. Rik's VM patches that make the replacement policy tend towards not evicting executable pages help a lot in this regard in a fairly obvious way. Anyway, we're well out of the realm of relevance to f-devel. Feel free to ping me off-list about it if you'd like. Kyle -- fedora-devel-list mailing list fedora-devel-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-devel-list
