Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Stefan Kost]

Am 04.02.2011 02:11, schrieb Niels Mayer:
> On Wed, Feb 2, 2011 at 12:48 PM, Stefan Kost  > wrote:
> 
> Sorry for the very late reply. One thing to mention is that the CPU load 
> for
> pulse is going wown quite a bit when plugin headphones. Otherwise it runs 
> some
> extra processing needed to accomodate the speakers.
> 
> 
> Since there is a lot of extra audio processing going on, would employing a
> library like "nova-simd", announced below, help out with the situation -- e.g.
> writing the handset speaker-equalization and some of the audio mixing
> functionality using more efficient instructions? What looks interesting is the
> claimed support for both ARM Neon, Altivec, and  Atom: See
> http://tim.klingt.org/code/projects/nova-simd/repository and forwarded
> announcement below.

Interesting. Need to take a closer look. Please note that pulseaudio is written
in C. Orc might be better suitable for PA also.

Stefan

> 
> 
> 
> Regarding pulseaudio,
> in 
> http://code.google.com/p/ytd-meego/wiki/CitizenJournalismWithYoutubeDirectForMeego#Pulseaudio:_Connecting_Audio_Subsystem,_Microphone_and_Speakers
>  -- Note
> all the "ins and outs" back and forth between kernel and user space.
> 
> Architecturally, the system seems inverted, especially since you have to go 
> back
> out to ALSA for bluetooth. Why not keep it all in kernel space?? 
> 
> Or better yet: employ analog mixing and multiple DAC's -- only one or two
> needing to be "hi-fi" and the others for notifications and telephony can be
> quite cheap low-rate, low-accuracy DAC's and ADC's. Yes analog mixing and
> routing is hard to do in a mixed signal environment; but is the power
> consumption of a few DACs and op-amps even close to that of needing a  
> fraction
> of the CPU running constantly just to do what a few resistors and an op-amp 
> can
> do? To say nothing about the potential sound quality improvement of not 
> needing
> to resample any of the mixed outputs, nor worry about synchronizing or locking
> rates between different subsystems, and all the other benefits of analog
> summing: http://emusician.com/mag/emusic_sum_tracks/ ...  (hopefully the
> flamewars that will ensue about this controversial topic will cancel out the
> flamewars on pulseaudio  :-) ) .
> 
> What I'm proposing for the low-power/handset world isn't that far off from the
> audio system needed  back when CPU's had computational power that made digital
> mixing for notifications, telephony, or music playback unthinkable: the analog
> mixing built into the http://en.wikipedia.org/wiki/AC'97
>  spec (and associated hardware), but 
> updated
> for the handset world. I'm sure that's not going to happen anytime soon, but I
> can dream?
> 
> ...
> 
> Anyways, here's the announcement:
> 
>  http://music.columbia.edu/pipermail/andraudio/2011-February/98.html
> ..
> [andraudio] nova-simd suite updated for arm neon
> Dan Stowell
> Thu Feb 3 04:48:36 EST 2011
> 
> Hi -
> 
> "nova-simd" is a C++ header-only template library for taking advantage
> of SIMD instructions in the kind of DSP processes used in audio
> processing (i.e. not just basic stuff like vectorised mul+add and loop
> unrolling, but also some more audio-specific things like ramps,
> distortions).
> 
> It's created by Tim Blechmann and it lives here:
> http://tim.klingt.org/code/projects/nova-simd/repository
> 
> It was originally implemented for SSE and SSE2 - we use it in
> SuperCollider and it gives us massive performance improvements on
> intel chips. We've just added NEON (and Altivec) implementations too,
> so we should be able to boost performance on android and ios.
> 
> The ARM NEON support is new and doesn't yet implement all the
> speedups, but should be useful for anyone wanting to implement C++ DSP
> for Android devices with SIMD speedups - especially if you're thinking
> about cross-platform code since the API is implemented for other chips
> too.
> 
> Dan
> .
> 
> Niels
> http://nielsmayer.com

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Niels Mayer]

On Wed, Feb 2, 2011 at 12:48 PM, Stefan Kost 
 wrote:
>
> Sorry for the very late reply. One thing to mention is that the CPU load
> for
> pulse is going wown quite a bit when plugin headphones. Otherwise it runs
> some
> extra processing needed to accomodate the speakers.


Since there is a lot of extra audio processing going on, would employing a
library like "nova-simd", announced below, help out with the situation --
e.g. writing the handset speaker-equalization and some of the audio mixing
functionality using more efficient instructions? What looks interesting is
the claimed support for both ARM Neon, Altivec, and  Atom: See
http://tim.klingt.org/code/projects/nova-simd/repository and forwarded
announcement below.



Regarding pulseaudio, in
http://code.google.com/p/ytd-meego/wiki/CitizenJournalismWithYoutubeDirectForMeego#Pulseaudio:_Connecting_Audio_Subsystem,_Microphone_and_Speakers
-- Note
all the "ins and outs" back and forth between kernel and user space.

Architecturally, the system seems inverted, especially since you have to go
back out to ALSA for bluetooth. Why not keep it all in kernel space??

Or better yet: employ analog mixing and multiple DAC's -- only one or two
needing to be "hi-fi" and the others for notifications and telephony can be
quite cheap low-rate, low-accuracy DAC's and ADC's. Yes analog mixing and
routing is hard to do in a mixed signal environment; but is the power
consumption of a few DACs and op-amps even close to that of needing a
 fraction of the CPU running constantly just to do what a few resistors and
an op-amp can do? To say nothing about the potential sound quality
improvement of not needing to resample any of the mixed outputs, nor worry
about synchronizing or locking rates between different subsystems, and all
the other benefits of analog summing:
http://emusician.com/mag/emusic_sum_tracks/ ...  (hopefully the flamewars
that will ensue about this controversial topic will cancel out the flamewars
on pulseaudio  :-) ) .

What I'm proposing for the low-power/handset world isn't that far off from
the audio system needed  back when CPU's had computational power that made
digital mixing for notifications, telephony, or music playback unthinkable:
the analog mixing built into the http://en.wikipedia.org/wiki/AC'97 spec
(and associated hardware), but updated for the handset world. I'm sure
that's not going to happen anytime soon, but I can dream?

...

Anyways, here's the announcement:

 http://music.columbia.edu/pipermail/andraudio/2011-February/98.html
..
[andraudio] nova-simd suite updated for arm neon
Dan Stowell
Thu Feb 3 04:48:36 EST 2011

Hi -

"nova-simd" is a C++ header-only template library for taking advantage
of SIMD instructions in the kind of DSP processes used in audio
processing (i.e. not just basic stuff like vectorised mul+add and loop
unrolling, but also some more audio-specific things like ramps,
distortions).

It's created by Tim Blechmann and it lives here:
http://tim.klingt.org/code/projects/nova-simd/repository

It was originally implemented for SSE and SSE2 - we use it in
SuperCollider and it gives us massive performance improvements on
intel chips. We've just added NEON (and Altivec) implementations too,
so we should be able to boost performance on android and ios.

The ARM NEON support is new and doesn't yet implement all the
speedups, but should be useful for anyone wanting to implement C++ DSP
for Android devices with SIMD speedups - especially if you're thinking
about cross-platform code since the API is implemented for other chips
too.

Dan
.

Niels
http://nielsmayer.com
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Stefan Kost]

Am 28.12.2010 00:26, schrieb Nick Copeland:
>> > Now PA does use
>> > quite a lot of CPU on the N900 - the +/-2% it requires on my laptop
>> > translates into about 25% on Maemo, this really is quite an overhead and
>> > as far as I can tell it does not change with sampling rate (I get the
>> > same overhead with 48kHz as with 44.1kHz although I will retest that).
>>
>> btw, one thing to watch out when doing measurements is the CPU
>> frequency... N900 uses very aggressive cpufreq and even with heavy audio
>> processing, the CPU is not necessarily running at full speed (so even if
>> top shows 25%, that doesn't necessarily mean that only 75% processing
>> capacity is left for apps). But this is really only a problem when making
>> measurements (the cpufreq is boosted automatically when needed).
> 
> The CPU load for pulseaudio does not appear to change much with sampling
> rate, taking 24kHz, 48kHz gives a very similar profile but you mention a bug
> report that notes that fact anyway. Bristol does have a very different 
> footprint
> depending on sampling rate and filter selection.
> 
> The code will attempt to set the CPU freq to 600MHz when it starts, it does
> not necessarily need these cycles but it helps to reduce underruns. Pulse
> now rolls along at about 8% to 10% CPU for any rates I have tried and bristol
> will take about 8% at 24kHz and double that (17%) for 48kHz.
> 
>> There should be no resampling happening (if you use 48kHz), so it
>> should not be that. But if src is used in your case, N900 uses speex src
>> (speex-fixed-2) optimized for arm NEON (see Siarhei's comment in bug for
>> info and links:
>> https://test.maemo.org/testzilla/show_bug.cgi?id=5794#c10 ).
> 
> This refers to the same data as above, ie, the resampling algorithm is not
> where the CPU cycles are being spent. The report discusses the same results
> as above but comparing 44.1kHz to 48kHz.
>  
>> Don't take this too seriously, but I've personally found it a bit
>> suprising how much negative feedback there has been about the audio mixer
>> CPU cycles. In phones (versus desktop/laptops), some processing is needed
>> as audio quality (especially for voice calls) is a closely followed aspect
>> (it is benchmarked by various parties and various organizations have
>> detailed requirements concerning it).
> 
> It is sometimes difficult to understand the added value of a given process.
> My first reply noted the fact that a complete synth voice with osc/mix/fil/env
> and all its internal processing overhead had a lower footprint than the audio
> daemon. It is easy to fall into 'Sendmail Syndrome' implying that the process
> takes a lot of time, effort (in sendmails case configuration as well) whilst
> not evidently bringing much to the table.

Sorry for the very late reply. One thing to mention is that the CPU load for
pulse is going wown quite a bit when plugin headphones. Otherwise it runs some
extra processing needed to accomodate the speakers.

Stefan
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Nick Copeland]

> > Now PA does use 
> > quite a lot of CPU on the N900 - the +/-2% it requires on my laptop 
> > translates into about 25% on Maemo, this really is quite an overhead and 
> > as far as I can tell it does not change with sampling rate (I get the 
> > same overhead with 48kHz as with 44.1kHz although I will retest that).
> 
> btw, one thing to watch out when doing measurements is the CPU 
> frequency... N900 uses very aggressive cpufreq and even with heavy audio 
> processing, the CPU is not necessarily running at full speed (so even if 
> top shows 25%, that doesn't necessarily mean that only 75% processing 
> capacity is left for apps). But this is really only a problem when making 
> measurements (the cpufreq is boosted automatically when needed).

The CPU load for pulseaudio does not appear to change much with sampling
rate, taking 24kHz, 48kHz gives a very similar profile but you mention a bug
report that notes that fact anyway. Bristol does have a very different footprint
depending on sampling rate and filter selection.

The code will attempt to set the CPU freq to 600MHz when it starts, it does 
not necessarily need these cycles but it helps to reduce underruns. Pulse 
now rolls along at about 8% to 10% CPU for any rates I have tried and bristol
will take about 8% at 24kHz and double that (17%) for 48kHz.

> There should be no resampling happening (if you use 48kHz), so it
> should not be that. But if src is used in your case, N900 uses speex src 
> (speex-fixed-2) optimized for arm NEON (see Siarhei's comment in bug for
> info and links:
> https://test.maemo.org/testzilla/show_bug.cgi?id=5794#c10 ).

This refers to the same data as above, ie, the resampling algorithm is not 
where the CPU cycles are being spent. The report discusses the same results
as above but comparing 44.1kHz to 48kHz.
 
> Don't take this too seriously, but I've personally found it a bit 
> suprising how much negative feedback there has been about the audio mixer 
> CPU cycles. In phones (versus desktop/laptops), some processing is needed 
> as audio quality (especially for voice calls) is a closely followed aspect 
> (it is benchmarked by various parties and various organizations have 
> detailed requirements concerning it).

It is sometimes difficult to understand the added value of a given process. 
My first reply noted the fact that a complete synth voice with osc/mix/fil/env
and all its internal processing overhead had a lower footprint than the audio
daemon. It is easy to fall into 'Sendmail Syndrome' implying that the process
takes a lot of time, effort (in sendmails case configuration as well) whilst
not evidently bringing much to the table.

I was actually pleasantly surprised by how well the N900 managed to support
the app, the 8/16 percent CPU footprint is better than I was anticipating for
all the floating point activity here and in truth pulseaudio is also showing 
that
same rough load profile with respect to the two very different systems.

The point you make in your email is that pulse is bringing features to the table
here and that they invariably come with some cost. It doesn't appear that it is
unexpected load though.

If anybody wants to test bristol on the Maemo/N900 it can be found here:

http://sourceforge.net/projects/bristol/files/bristol/0.60/bristol-0.60.8.deb.tar.gz/download
http://bristol.sourceforge.net/news.html

There are notes in the news page regarding how to install the .deb, you need 
root access installed for the dpkg and as the file had to be tarred and gzipped
to be accepted by sfdc you have a couple of extra steps either on the phone
or on your PC prior to the install.

Kind regards, nick

  ___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Niels Mayer]

I'm replying out of order and still digesting all the messages... but
first wanted to thank all the Nokia engineers and architect willing to
come here and discuss their experience with the N900 product
multimedia issues publicly. This wouldn't be happening with Apple, or
Android, so I wanted to express my appreciation for your commitment to
open-ness,  open-source, and open discussion.

On Sun, Dec 19, 2010 at 12:43 PM, Kai Vehmanen  wrote:
> And another btw, in case someone has missed this, do check Jyri Sarha's
> slideset presented at plumber's conf 2009:
> http://linuxplumbersconf.org/2009/slides/Jyri-Sarha-audio_miniconf_slides.pdf
>
> E.g. slide 15 has a good diagram of how the pipelines are connected.
>
> Some things to try:
>  - use an output low overehead output (e.g. headset)
>  - use srate that matches hw rate (48kHz in this case) ->no SRC is
>    involved

These are interesting suggestions and that is a very good link
explaining the issues.  Also, I was wondering how the n900 speakers
actually managed to sound so good while being minuscule -- so you're
saying a finite impulse response adjustment has been made? But
wouldn't that be different "mid air" versus "on kickstand" versus flat
on table (boundary effect)? [2.0 version feature: haptic adjustment of
FIRs] And that processing overhead can be reduced by plugging-in
headphones or activating stereo output?

..

So interestingly, in [1] one of the main issues is latency -- the
exact same one I raised as being important for "creative" multimedia.
But clearly , it's also important for day-to-day tasks on the handset.
The quotes below are almost as if pulseaudio is being used in a
"jack-like" fashion, with small buffers, rather than the usual
power-efficient large buffers. But that also means pulseaudio is
consuming more CPU cycles at high-prio, competing against shorter
emptying buffers. Does the "Policy Framework" orchestrate switching
pulseaudio's buffer-lengths  "on the fly" in order to provide
low-latency when needed for a call, and more power-efficient
performance for streaming audio playback, such as listening to music
or watching a video?
...

[1] 
http://linuxplumbersconf.org/2009/slides/Jyri-Sarha-audio_miniconf_slides.pdf

Cellular Call Latency Challenge
• Cellular modem air interface alone causes a lot of latency
• GSM and 3G audio frame size is 20 ms
• Cellular frame timing is ruled by base station
• Timing of 20ms frames change in cellular hand over
• Simple buffering between cellular modem and audio codec adds
at least 20 ms latency to each direction

Minimize Cellular Call Latency
• Run ALSA with two 5 ms fragments
 This is doable even on ARM Linux with real-time priority
 DMA buffering delay is 5ms each direction
• Synchronize up link audio buering with Cellular Modem
 Cellular Modem sends up-link timing adjustment messages
 Align up-link buering according to messages
 Change UL timing with 5 ms granularity
• Synchronize down link audio buering with Cellular Modem
 Keep tight buffer management to minimize latency

...
... Acoustic Echo Cancellation 
...

• Good acoustic echo path modeling for transient signals is
possible only with proper time alignment of the reference loop

Accurate feedback loop timing for AEC
• Accurate playback and capture latencies are needed for time
alignment of echo reference and mic signal
• Latency functions of ALSA do not work well with
doing DMA block transfers
• Solution: Implement ALSA sink latency functions based on
snd_pcm_htimestamp()


-- Niels
http://nielsmayer.com
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Kai Vehmanen]

Hi,

On Sun, 19 Dec 2010, Nick Copeland wrote:

That was a very interesting post, will be keeping it. Now PA does use 
quite a lot of CPU on the N900 - the +/-2% it requires on my laptop 
translates into about 25% on Maemo, this really is quite an overhead and 
as far as I can tell it does not change with sampling rate (I get the 
same overhead with 48kHz as with 44.1kHz although I will retest that).


btw, one thing to watch out when doing measurements is the CPU 
frequency... N900 uses very aggressive cpufreq and even with heavy audio 
processing, the CPU is not necessarily running at full speed (so even if 
top shows 25%, that doesn't necessarily mean that only 75% processing 
capacity is left for apps). But this is really only a problem when making 
measurements (the cpufreq is boosted automatically when needed).


You can use standard kernel interfaces to limit the CPU frequencies 
(although for sake of your battery, I'd stick to the defaults for all 
normal usage).


And another btw, in case someone has missed this, do check Jyri Sarha's 
slideset presented at plumber's conf 2009:

http://linuxplumbersconf.org/2009/slides/Jyri-Sarha-audio_miniconf_slides.pdf

E.g. slide 15 has a good diagram of how the pipelines are connected.

Some things to try:
  - use an output low overehead output (e.g. headset)
  - use srate that matches hw rate (48kHz in this case) ->no SRC is
involved


I am assuming, possibly incorrectly, that it uses Lennart's correct resampling
algorithm. Is this the case? If so is there any way an application can request


There should be no resampling happening (if you use 48kHz), so it
should not be that. But if src is used in your case, N900 uses speex src 
(speex-fixed-2) optimized for arm NEON (see Siarhei's comment in bug for

info and links:
https://test.maemo.org/testzilla/show_bug.cgi?id=5794#c10 ).


Don't take this too seriously, but I've personally found it a bit 
suprising how much negative feedback there has been about the audio mixer 
CPU cycles. In phones (versus desktop/laptops), some processing is needed 
as audio quality (especially for voice calls) is a closely followed aspect 
(it is benchmarked by various parties and various organizations have 
detailed requirements concerning it). And you can't just solve this by 
using a bit better components as the devices are physically so small (and 
packed with stuff), are often used in very audio-hostile/noisy 
environments, all that makes the acoustics design anything but easy 
(and fine-tuning with the help of SW always helps). E.g. the 10" laptop 
I'm writing this on has speakers the size of almost half the length of a 
phone, and the netbook speakers still sound pretty bad. Maybe the laptop 
could use a bit of DSP help as well to get more out of the same hw... 
(BruteFIR, anyone..). :P


And OTOH, if the whole thing had been hid on a HW blob (and as is common, 
with no possibility to run 3rd party software on it, nor otherwise control 
it), roughly same amount of cycles would still be spent, although on 
another core (and not showing up in Linux 'top' output). For overall 
battery consumption this can (but not of course in all cases) be worse as 
the Linux CPU is anyways waking up, so it can be actually more efficient 
to do the processing there as well (the caches are hot and CPU already 
powered, so doing a bit more work per wakeup is in fact quite efficient). 
Now that N900 does this in a more open way, hordes of people are screaming 
bloody murder about wasting CPU on processing samples! ;D


Oh well, I do understand people want all the CPU cycles they can for apps, 
and this is of course perfectly reasonable (and thus don't take this too 
seriously).



On a more serious note, the key take away (especially to meego-handset 
folks) is that this particular worry has very little to do with 
Pulseaudio. You can do a MeeGo device without any of the abovementioned 
processing, use separate DSPs, and still use Pulseaudio on top as a 
frontend mixer (with very little overhead). Bypassing  PA completely 
should be possible as well, but then the resource framework enforcement 
points need to be extended to cover the hw mixers (so that the handset 
behaves as expected with regards to routing, application priority and 
volume policies in effect -> you don't want "new email" tones to your 
video recording, and so forth).

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Nick Copeland]

> so as mentioned in other replies, this is not really pulseaudio to blame. 
> On N900 there is some fixed processing that must be applied to all streams 
> (and there are different pipelines for different uses and routes, so it's 
> not always constant). With N900 this code is now in PA (as it's the system 
> mixer, there's no hw mixer under the hood). The load is smaller e.g. when 
> you use earpiece/headset (versus speaker), and also it should be lower if 
> you have the apps use the hw rate (48kHz in N900 case). You can challenge 
> the N900 design, but none of this is really specific to Pulseaudio.
[snip}

That was a very interesting post, will be keeping it. Now PA does use quite
a lot of CPU on the N900 - the +/-2% it requires on my laptop translates into 
about 25% on Maemo, this really is quite an overhead and as far as I can tell
it does not change with sampling rate (I get the same overhead with 48kHz
as with 44.1kHz although I will retest that).

I am assuming, possibly incorrectly, that it uses Lennart's correct resampling
algorithm. Is this the case? If so is there any way an application can request 
an alternative? To get bristol to work on the N900 without gobbling up masses
of CPU required large period sizes, relatively large preloading of buffers with 
added latency, downsampling to 24kHz, no resampling filters. The last two 
reduce the audio quality but prevent it from guzzling battery like there were 
no tomorrow. If PA is still using the intense resampling to maintain quality 
then there is an issue here - there would be no point is quality resampling as 
the signal has already been degraded.

Regards, nick.
  ___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Kai Vehmanen]

Hi,

On Sat, 18 Dec 2010, Niels Mayer wrote:


It has decent performance on maemo and appears to use pulseaudio,
which eats 1/3 of the CPU of the 'sunvox' process. The sunvox

[...]

 PID  PPID USER STAT   RSS %MEM %CPU COMMAND
1916  1162 user S 6388  2.5 35.1 /usr/bin/sunvox
 825 1 pulseR <   3812  1.5 12.1 /usr/bin/pulseaudio --system


so as mentioned in other replies, this is not really pulseaudio to blame. 
On N900 there is some fixed processing that must be applied to all streams 
(and there are different pipelines for different uses and routes, so it's 
not always constant). With N900 this code is now in PA (as it's the system 
mixer, there's no hw mixer under the hood). The load is smaller e.g. when 
you use earpiece/headset (versus speaker), and also it should be lower if 
you have the apps use the hw rate (48kHz in N900 case). You can challenge 
the N900 design, but none of this is really specific to Pulseaudio.


The N900 audio design in fact gives quite a lot of responsibility to Linux 
side (perhaps an interesting comparison is the Palm Pre as it also uses 
Pulseaudio). This has some cost, but also a great deal of flexibility (in 
terms of what can be implemented, but also for work flows as you can 
devel&debug the whole audio system with same Linux development tools). The 
more traditional approach is to put all of this logic to a separate HW 
block, but then the Linux side has even less control and access to what 
happens to the audio streams. In these cases, Pulseaudio becomes just a 
frontend mixer for streams originating from/to Linux apps. In this sense 
N900 allows you much closer access to the HW, and there is more potential 
for really low-latency uses with this architecture. I'm personally 
thrilled of the potential of this approach (while not claiming the 
implementation is perfect yet).



Doing an "ls -lR /" in a remote xterm (over SSH) results in some audio
glitching, but no "desynchronization" where the audio just stops
playing.


This is a real problem for sure.

I believe the cause for the glitches is that sunvox's audio thread is not 
run under SCHED_FIFO/RR. The real/nonrealtime sync point really should be 
in application's hands (like it is by default in JACK apps). If Pulseaudio 
alone tries to handle this (trying to buffer audio from/to nonrealtime 
apps), you will get huge latencies (and huge utilization of 
unswappable locked memory).


But the problem is not really Pulseaudio, but instead the mechanisms to 
allow/manage FIFO/RR. JACK of course provides a good system for setting 
the scheduling for clients, but the core problem is not really setting the 
scheduling policy, but how to manage and coordinate the system-wide impact 
of multiple real-time apps, and their possible conflicts.


When you build a JACK system, jackd and related apps take priority over 
everything else on the system (in real-time sense). Avoiding glitches for 
them is of utmost priority (UI can lag as long as audio runs without 
xruns). On a mobile device, there are various other important functions 
that need real-time (the exact list varies from product to product but 
there are always some), and which must work. So even if you use apps like 
Sunvox, you still want to be able to receive/make calls reliably, be 
certain that the device wakes you up in the morning, and is ready for 
quick camera shooting when the situation arises. So overall system 
reliability and responsiveness is really important. Having lots of 
uncoordinated (e.g. developed separately, not referring to quality or 
cases of possible malicious intent here) real-time apps is a tricky 
problem. It's not a new problem, agreed, but still a difficult to solve in 
the generic case.


Ideally we'd have more scheduling policy options in Linux for this (there 
have been no shortage of proposals, also from Nokia), but currently 
FIFO/RR are still the best options on the table for low-latency audio.


Now there are some building blocks in place already in MeeGo/Maemo (e.g. 
cgroups scheduling is used and there is a system for managing resources), 
but there are still gaps that cause trouble to apps. Currently the only 
way to achieve JACK-style low-latency is to write your app's audio thread 
as a PA module. Aside cgroups, real-time watchdogs could be used, and even 
with the risk of restarting past flamewars ;), PolicyKit is obviously one 
option as well to improve this area.


But to summarize, the core problems that still need addressing w.r.t. 
JACK-style low latency, are not really related to Pulseaudio (nor solved 
simply by switching to JACK). If MeeGo/Maemo would have used JACK, but had 
no managed system in place for passing FIFO/RR to apps, the only option to 
implement low-latency engines would be to write them as inprocess JACK 
apps. So you'd have the same exact problem as with Pulseaudio (and have 
many apps running nonrealtime despite their need for low-latency access to 
the mixer -> glitches in cases 

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Kai Vehmanen]

Hi,

On Sat, 18 Dec 2010, Niels Mayer wrote:


The list comprises a good number of people with expertise in both
pulseaudio; hopefully the Jack sound server authors, including Paul
Davis, will be willing to publicly share their perspectives on the
issues raised regarding the role of pulseaudio on a handset and  Linux
audio performance/latency/efficiency issues.


btw, just a note that there are lots of LAD/JACK regulars (including me) 
currently working with Maemo/MeeGo, so there's some (or actually quite a 
lot) of history to this discussion.


Here's my reply to Benno's mail (July this year) about this same topic on 
maemo-developers: 
http://lists.maemo.org/pipermail/maemo-developers/2010-July/027087.html


I think many (not all but many) issues you Niels raised in your original 
mail would happen with JACK as well:

  - Most of the CPU load attributed to Pulseaudio is actually
algorithms specific to the product (in-process clients in JACK
terms, so would show up with jackd as well).
  - Bugs in kernel drivers that cause scheduling spikes (your WLAN
scan example) would affect jackd just as badly (bear in mind
that your average mobile phone has a huge number of hw drivers,
so there's a lot of kernel code to go through for fine-tuning);
The Maemo5 N900 firmware should fair much better (it has gone
through more fine-tuning).. This is very familiar to all of us
building machines to run jackd. Even one badly behaving driver
can ruin the whole thing, and hunting these down is not a small
task.
  - Locking the sample-rate is not specific to Pulseaudio. It's
more related to the problem of supporting multiple, totally
independent clients; e.g. you could lock the hw to 8000 for calls,
but if then video playback is started, while the call is still
ongoing, do you a) stick to 8000Hz for the whole video as well,
b) have a glitch as you need to reprogram the hw to different rate,
or c) do you do the common thing and run at a common backend rate
(like 48000) and resample some of the client streams.

Agreed, JACK has many strong points that apply to MeeGo context as well, 
especially in providing a good API for out-of-process clients to get 
low-latency access to the audio engine. You can do the same with 
Pulseaudio, but it's more complicated and more prono to errors (e.g. the 
QAudio recording example Benno mentioned in the above thread). But then 
OTOH, Pulseaudio has many strong points as well. Both projects can be 
extended/developed. Of course here one important factor is what the 
projects themselves want to do... and here Pulseaudio has a much more 
explicit focus on wanting to solve the wider audio-on-the-desktop problem, 
and it has had that for a long time.

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Marco Ballesio]

Hi,

I'm now subscribed to the linux-audio ML, so more people will be
happily spammed directly from my mailbox... some notes below.

On Sat, Dec 18, 2010 at 8:33 PM, Niels Mayer  wrote:
> FYI, here's an example of the kind of app that needs to have good audio
> performance on a handset:
> http://www.youtube.com/watch?v=bwqflVX5oNo
> http://www.warmplace.ru/soft/sunvox/
> ( 
> http://lists.linuxaudio.org/pipermail/linux-audio-user/2010-December/074828.html
> )

Good example, even though I prefer Buzztard ;) . Dunno if the latter
runs on embedded devices though.

I also wrongly wrote we were on the IVI mailing list, while it
actually was the Handset one, where it's more likely to have audio
editing applications.

>
> It has decent performance on maemo and appears to use pulseaudio,
> which eats 1/3 of the CPU of the 'sunvox' process. The sunvox
> application appears to have a UI thread and a worker thread each
> consuming about 1/2 of the 35% CPU load of the app.

another issue with audio servers running on an "embedded" device is
the particular care we need to give to the memory usage: as the
swapper may not have pity when time comes to go and find some system
memory, the latencies introduced by the operation on the real-time
process could suspend it for a while. And if you have plenty of
modules loaded, memory-locking the process is a mere chimera.

The audible effect, as usually the DMA keeps on running from the audio
chip data bus, is that the latter continues transferring what's stored
there as if it was a ringbuffer, with what I love to call the
"Woodland Critters" (woodpecker, squirrel-eating-acorn, etc..) effect.

Here, we could still argue about the necessity of virtual memory on an
"embedded" device, but imho it comes together with the one to give
users an experience equivalent to the one they've on their desktop,
the cost of SDRAMS wrt NAND flashes and the delta in energy
consumption...

>
> Mem: 238432K used, 7108K free, 0K shrd, 3396K buff, 67580K cached
> CPU: 52.2% usr  7.8% sys  0.0% nice 39.8% idle  0.0% io  0.0% irq  0.0% 
> softirq
> Load average: 0.99 0.45 0.16
>  PID  PPID USER     STAT   RSS %MEM %CPU COMMAND
>  1916  1162 user     S     6388  2.5 35.1 /usr/bin/sunvox
>  825     1 pulse    R <   3812  1.5 12.1 /usr/bin/pulseaudio --system
> --high-priority
>  897   730 root     S <  16524  6.7  9.6 /usr/bin/Xorg -logfile
> /tmp/Xorg.0.log -logverbose 1 -nolisten tcp -noreset -s 0 -core
>
> Doing an "ls -lR /" in a remote xterm (over SSH) results in some audio
> glitching, but no "desynchronization" where the audio just stops
> playing.

The audio glitching may actually come from the effect I exposed above
(from "top", system memory is loaded at more than 50%). The Resource
Policy Framework would help with this (it's not only handling audio
routing, but also the system memory available for processes, forcing
some to swap more often than others) but it can't do miracles ;).

The apparently high CPU load may be due to many causes, like the ones
I wrote about in my previous email. Also, setting a too low latency on
the pulseaudio client tends to increase the CPU usage. In my mind I
explained it as the fact that low latencies require shorter time
windows which, after a Fourier transformation, imply a wider frequency
range to be processed per time unit.

Regards

>
> -- Niels.
>
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Niels Mayer]

FYI, here's an example of the kind of app that needs to have good audio
performance on a handset:
http://www.youtube.com/watch?v=bwqflVX5oNo
http://www.warmplace.ru/soft/sunvox/
( 
http://lists.linuxaudio.org/pipermail/linux-audio-user/2010-December/074828.html
)

It has decent performance on maemo and appears to use pulseaudio,
which eats 1/3 of the CPU of the 'sunvox' process. The sunvox
application appears to have a UI thread and a worker thread each
consuming about 1/2 of the 35% CPU load of the app.

Mem: 238432K used, 7108K free, 0K shrd, 3396K buff, 67580K cached
CPU: 52.2% usr  7.8% sys  0.0% nice 39.8% idle  0.0% io  0.0% irq  0.0% softirq
Load average: 0.99 0.45 0.16
  PID  PPID USER STAT   RSS %MEM %CPU COMMAND
 1916  1162 user S 6388  2.5 35.1 /usr/bin/sunvox
  825 1 pulseR <   3812  1.5 12.1 /usr/bin/pulseaudio --system
--high-priority
  897   730 root S <  16524  6.7  9.6 /usr/bin/Xorg -logfile
/tmp/Xorg.0.log -logverbose 1 -nolisten tcp -noreset -s 0 -core

Doing an "ls -lR /" in a remote xterm (over SSH) results in some audio
glitching, but no "desynchronization" where the audio just stops
playing.

-- Niels.
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Niels Mayer]

Marco -- many thanks for your reply -- I've taken the liberty of
CC'ing  this informative reply to the Linux-Audio-Developers list
(your CC was not posted as you're not a subscriber:
http://lists.linuxaudio.org/pipermail/linux-audio-dev/2010-December/thread.html
).

The list comprises a good number of people with expertise in both
pulseaudio; hopefully the Jack sound server authors, including Paul
Davis, will be willing to publicly share their perspectives on the
issues raised regarding the role of pulseaudio on a handset and  Linux
audio performance/latency/efficiency issues.

Here's a link to Marco's original  post to meego-handset mailing list,
forwarded below:
http://lists.meego.com/pipermail/meego-handset/2010-December/90.html

-- Forwarded message --
From: Marco Ballesio 
Date: Sat, Dec 18, 2010 at 4:42 AM
Subject: Re: Meego pulseaudio "compliance" and "enforcement" (was Re:
[Meego-handset] Enabling Speakerphone)
To: Niels Mayer 
Cc: meego-hand...@lists.meego.com, Linux Audio Developers
 [...]

Hi,

sorry for the late reply but, whew, this was really long, definitely
more than what my poor brain can handle in a single bunch ;). Maybe we
could split this into a "Resource Policy Framework" and "Pulseaudio on
Embedded" couple of threads.

As a general comment on PA, it may appear odd but when I started
working with it I shared 100% of your considerations. Working with it
(not ON it), I started realising that, ok, it may not be perfect (and
which software actually is?) but it brings definitely more benefits
than troubles to the system. See my notes below for more details.

..snip..

>
> Hopefully there's more than just source-code to describe the policies
> and enforcement.

Maybe you've read in some other of my posts here and there, I'm trying
to get a Resource Policy Framework wiki somewhere. In the meanwhile,
documents are scarce but present:

- The MeeGo conference presentation:

http://conference2010.meego.com/session/policy-framework-flexible-way-orchestrate-multiple-functionalities-meego-devices

Does not say too much, but it comes with some slides and many "h"
(keepalive signals).

- The resource policy application developer's manual:
http://wiki.meego.com/images/Meego-policy-framework-developer-guide.pdf

Mysteriously, it's not linked or referred to anywhere.. at least, now
it's on this thread.

- The source code: it may twist more than one nose but, at the end, I
always like to check in the sources before/after Googling. Documents
may be obsolete (who knows?) or expose a partial view. Code can't ;).
Not that I want to say with this that we don't need more documents
about the Resource Policy Framework...

> Is there a design document stating the
> "organizational" or "legal" or "human" reasons for said policies and
> enforcements?

My pointers of above may begin to satiate your hunger (appetisers?).

>
> And, as a developer, where is the documentation for the appropriate
> layer to use, the appropriate API, etc -- e.g. for the original
> question that started this thread -- how to switch on/off speakers,
> FM-radio transmitter,  etc on handset. ( cf my original reply
> http://lists.meego.com/pipermail/meego-handset/2010-December/66.html
> )

Unfortunately, this is still in the "dark side" of the documentation,
which is currently application-centric. Hopefully one day we'll have
an "Adaptation developer's guide", in the meanwhile, a good starting
point may be checking the sources, the N900/Meego ruleset and posting
questions here. If it's not enough, we may set up an IRC channel (if
there's enough interest)...

>
> Something high-level like the following Maemo document would be very
> helpful -- but I have been unsuccessful finding such documentation for
> Meego: 
> http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Architecture/Multimedia_Domain

See my notes above about the project wiki. Your pointers are
definitely good hints, thanks.

>
> Also, why not use ALSA use-case-manager:
> http://opensource.wolfsonmicro.com/node/14
> UCM appears to be scheduled for other distros "handset" work:
> https://wiki.ubuntu.com/Specs/N/ARM/AlsaScenarioManager

this project covers the audio-specific cases: for what I understood it
might very well be used as an ALSA enforcement point in the Framework
so, more than compete with Resource Policy it would be complementary
in case you don't really want/need to use a sound server. BTW, I could
not find so much info, maybe you can help me to retrieve:

- more than just source-code to describe how it works.
- the developer's documentation for the appropriate layer to use, etc.
- something like:
http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Architecture/Multimedia_Domain

..snip..

now, the pulseaudio cases. First of all, please note I'm not the best
advocate for it, btw my 0.05 €.

Posting some extracts of your comments to the PA mailing list will
definitely give you a better insight for the internals of the server.

>

[LAD] Meego pulseaudio "compliance" and "enforcement" (was Re: [Meego-handset] Enabling Speakerphone)

[Niels Mayer]

On Tue, Dec 14, 2010 at 1:31 PM, Marco Ballesio
> Please check here:
>
> http://meego.gitorious.org/maemo-multimedia/pulseaudio-policy-enforcement/trees/master/src
> to get a few more hints on the subject.

Hopefully there's more than just source-code to describe the policies
and enforcement. Is there a design document stating the
"organizational" or "legal" or "human" reasons for said policies and
enforcements?

And, as a developer, where is the documentation for the appropriate
layer to use, the appropriate API, etc -- e.g. for the original
question that started this thread -- how to switch on/off speakers,
FM-radio transmitter,  etc on handset. ( cf my original reply
http://lists.meego.com/pipermail/meego-handset/2010-December/66.html
)

Something high-level like the following Maemo document would be very
helpful -- but I have been unsuccessful finding such documentation for
Meego: 
http://wiki.maemo.org/Documentation/Maemo_5_Developer_Guide/Architecture/Multimedia_Domain

Also, why not use ALSA use-case-manager:
http://opensource.wolfsonmicro.com/node/14
UCM appears to be scheduled for other distros "handset" work:
https://wiki.ubuntu.com/Specs/N/ARM/AlsaScenarioManager

> the n900 Resource Policy enforcement points were directly interacting
> with ALSA. As Krisztian pointed out, it's no more necessary (and at
> the limit it may be dangerous for your system's health) to do so
> within MeeGo, as the pulseaudio ports can elegantly handle the whole
> thing.

Potential to blow out my handset speakers duly noted (the n900 goes
http://en.wikipedia.org/wiki/Up_to_eleven !), however, I wasn't
advocating playing around with the ALSA layer indiscriminately, in
fact, I specifically stated:
> the "amixer" results of meego indicate an equally complicated
> soundchip; where random hacking could render your system somewhat useless... 
> is there documentation on all these values?:
> http://nielsmayer.com/meego/n900-card0-amixer.txt .
> The only chip mentioned by name is
> http://focus.ti.com/lit/ds/symlink/tpa6130a2.pdf "TPA6130A2 Headphone"



Regarding my old nemesis, pulseaudio ( http://tinyurl.com/2976vu6
== 
http://old.nabble.com/uninstall-pulseaudio-to-increase-audio-app-stability-across-updates-(was-Re:-yum-update)-td27759501.html#a27759501
), I think the only "elegant" thing about pulseaudio is that it's
bluetooth handling works for those that care about bluetooth;  given
the number of bug-reports and incompatibilities pulseaudio generates
in different distros, I'm not sure "elegant" is the right word

>From my position, as a multimedia/ALSA/linux-audio developer, having
to go through pulseaudio sounds like an all-around bad idea, and to
have "enforcement" or "compliance"  attached makes it sound even
worse. Tell me it ain't so!

There is a growing class of applications that do not want or need
pulseaudio around -- those using http://jackaudio.org/ .  When the
jack audio server launches, the first thing it does it use dbus to
disable pulseaudio. Is that also non compliant?

It seems inappropriate to preclude an entire class of application --
real-time "pro" audio and video apps that utilize the Jack audio
server to provide low-latency, tightly synchronized audio -- as needed
for modern multimedia creation and playback. Perhaps such applications
are a stretch for an OMAP3-class device, but given the many
audio/media apps listed in http://omappedia.org/wiki/PEAP_Projects ,
clearly OMAP4 and beyond might not be, even on a puny "handset." Of
course, those making such audio apps might sidestep pulseaudio
compliance/latency/inefficiency issues by using
http://opensoundcontrol.org/ and an external DAC (
http://gregsurges.com/tag/dac/ ).

Finally, it seems odd that in a "handset" environment, pulseaudio is
an absolute requirement. To me, it is just a waste of batteries, and a
terrible source of unnecessary context switching and interrupts during
audio playback. It's sort of like being forced to drive around in a
gas-guzzling, oversized sport-utility vehicle with 10 foot tires and 5
feet of ground clearance -- just to drive to the market or work on a
well-paved freeway on a summer's day -- even when one might prefer a
bicycle, motorcycle, sports-car, subway,  or whatever tool is best for
the job. Given that's the argument against Java/Android on the handset
(it's the SUV of languages/environments) it's unfortunate that
lighter-weight and less monolithic and more "unixy" solutions aren't
being pursued for audio on the Meego handset.

Take for example HD audio/video playback -- something where you need a
"sportscar" to get the latency and sample rate associated with the
audio stream while also performing HD decoding (e.g. 16bit/96K audio
is supported on omap3430 per
http://and-developers.com/device_information ). Pulseaudio typically
operates at a fixed rate and forces resampling to that rate, causing
an oft perceptible loss in fidelity. So in order to allow "digital
mixing" for n