Re: [LAD] cpu spikes

[Fokke de Jong]

thanks for all your input, I’ll try and summarize here.


> You're running Mint :-)  Lots of background bells and whistles there, lots of 
> things which will crop up and interfere, things you cannot disable or turn 
> off with absolute certainty.  If you want smooth power, you'll have to choose 
> more carefully.  My current SOP in more detail here:
> 
> http://lsn.ponderworthy.com/doku.php/choosing_a_linux_platform_for_live_synth 
> 
> 
> 
> 
> -- 
> Jonathan E. Brickman   j...@ponderworthy.com    
> (785)233-9977
> Hear us at http://ponderworthy.com  -- CDs and MP3 
> now available! 
> Music of compassion; fire, and life!!!


First of all, booting into console mode, rather than running the full blown 
desktop seemed to eliminate most of the problems, although it’s still not quite 
a stable as i’d like.
Also i don’t quite understand how all of that could interfere with my RT-thread.
This was going to try and install a more minimal system anyway, and don’t need 
a graphical environment for this, but during developments it’s kind of nice to 
have.

I still would like to see how far i can take this, and was really hoping i can 
continuously use 80-90% of all cpu cores without dropouts…
Is that realistic with a lowlatency kernel? 


> Do you lock all memory used by your RT threads ? 
> 
> If you don't and the system is configured for high swappiness
> [1] this sort of thing could happen. 
> 
> I'm routinely running big real-time convolution matrices without
> problems, so it's certainly possible. 
> 
> 
> [1]  >
> 
> -- 
> FA

I am not currently locking memory. I thought a had plenty of ram, as not to 
cause any swapping, but i guess its good practice to wire memory, so i will 
give it a try.

> 
> Bad kernel driver? WIFI drivers are known bad for things like this. An 
> interupt driver can block if it is designed badly. I found on one machine I 
> had to unload the the kernel module for my wifi as it actually created more 
> problems when I turned the power off to the tx than when it was on. (it seems 
> to me on my wifi, when it was turned on I got xruns every 5 seconds, but with 
> it turned off it was every half second or so... sounds very close to 0.6, 
> unloading the kernel module fixed it)
> 
> Cron should also be turned off, but that is probably not the problem here. 
> Cron runs super "nice" but there seem to be some things it does like packge 
> update that can cause problems too. I turn off cron while recording.
> 
> --
> Len Ovens

I don’t have a wireless on my machine, nor an nvidia card. just intel builtin 
graphics. This where my linux knowledge falls short, but If i don’t have that 
hardware, can I assume no drivers for it are loaded?

> 
> AFAIK, the important things are.
> 
> 1. Use a properly configured realtime patched kernel.
> 

lowlatency-kernel is not going to cut it?

I wasn’t really able to find to much info on the difference between the two, 
other than than the rt-kernel is a “step up” and hard realtime vs soft.
But nothing on how this is technically achieved


> 2. Set a high priority of the soundcard interrupt, something like 97 is
> a good value.  (If using a USB soundcard, set the priority of the
> interrupt servicing the USB hub instead).
> 

did that.
> 3. Run Jack with realtime and memlocking enabled and at a priority of
> 80.
> 
I’m not running jack but rather using alsa directly/

> 4. Make sure that you don't have any hardware/drivers that play havoc
> with your kernel scheduling.  some WIFI adapters, NVIDIA, etc comes to
> mind.
> 
> 5. Make sure that the system isn't suffering from SMI/NMIs which
> preempt the kernel and can take a long time to execute.  This can be
> done with hwlatdetect script in the rt-tests package.
> 
> 6. Use cyclictest from rt-tests to confirm that there are no latency
> spikes in how the kernel schedules threads.
> 
> Possibly hyperthreading, cpu power management, etc could cause
> problems, and I don't have experience with all hardware out there, but
> IME on modern Intel hardware this isn't a problem.
> 
I did actually find that hyperthreading had an impact, turing it of made every 
thing much more predictable.

> JACK2 also has a very nice profiling tool that can give a good idea
> about what is going on with the soundcard interrupt, clients, etc.
> 
> -- 
> 
>   Joakim


> Keep an eye on the interrupts while its all running, particularly
> Non-maskable interrupts. Try to correlate them with the 0.6 sec
> of the glitches if possible;
> 
> watch -n 0.1 cat /proc/interrupts
>  
> I've written up some of the checks I generally do, perhaps browse
> that to see if there's anything there that you could check?
> http://openavproductions.com/real-time-latency-tuning/ 
> 

Re: [LAD] cpu spikes

[Len Ovens]

On Mon, 25 Jan 2016, Joakim Hernberg wrote:


I suppose hyperthreading could be a potential pitfall, but personally I
see no problems with it with my audio workloads on my i7.


hyperthread is only a problem with jack latency under 64/2... even on an 
older single core P4. (at least in my testing)


--
Len Ovens
www.ovenwerks.net

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Len Ovens]

On Mon, 25 Jan 2016, Len Ovens wrote:


I am sure some will say that if rtirq doesn't help there is a bad driver...


Check the actual priorities that rtirq sets. It seems to me the last time 
I checked that if an irq is shared by a, b an c and rtirq is used to 
prioritize c to 90 for example, a and b will end up at 86 and 88 or 
something like that even though they should be 50. This was some time ago 
and may well have changed. In days of old I found even swapping the slots 
cards were plugged into made a difference, but I have the same cards 
backwards in the i5 I run now with no problem.


note: I use two audio cards, a delta66 and an audiopci. The delta has to 
be higher priority than the audiopci (which provides midi only) or I get 
xruns.



--
Len Ovens
www.ovenwerks.net

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Len Ovens]

On Mon, 25 Jan 2016, Fokke de Jong wrote:


  16:          0          0          0          0   IO-APIC   16-fasteoi   
madifx


Is this your audio interface on irq 16? If so why is it sharing an IRQ? 
Move it to a different slot maybe? If this is a PCI card and there is only 
one slot, I would suggest a different motherboard with more PCI slots. My 
personal experience with sharing IRQs has never been good even using rtirq 
to separate things out. One thing to try is in BIOS there is sometimes a 
setting that tells bios to set irqs or not. I have found setting it to not 
lets the kernel set it and the kernel does a better job. Also, some bios 
have a part where you can fix a PCI card to a irq, that may help.


I am sure some will say that if rtirq doesn't help there is a bad 
driver... OK. The thing to remember is that PCs are not built for low 
latency but high throughput. Most people find that high throughput makes 
for a "snappy" user experience. Low latency to most HW designers means 
30ms.


--
Len Ovens
www.ovenwerks.net
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Len Ovens]

On Mon, 25 Jan 2016, Jörn Nettingsmeier wrote:

sorry to hijack this thread, but: when enquiring about latency tuning, one 
frequently encounters hints like "disable cron", "disable indexing services", 
"disable this, disable that".


however, none of those alleged culprits run with real-time privileges or 
access driver or kernel code which does. so how can they be a problem (and 
disabling them part of the solution)? i'm asking because i've got my own 
anecdotal evidence that it *does* make a difference...


Yes, the big thing is that I see xruns just before something pops up 
saying "hey theres an upgrade available". Now as I have said, cron runs 
super "nice" and so anything that cron runs should be really low priority 
too. But time constraints are not just CPU access and time. I would think 
that the network driver even using the bus for a full 1500 bytes should 
not be a problem, but where does that data go? What priority is a disk 
access... and once it starts how big a chunk of data gets written and is 
it atomic? It does not seem to be memory related as I use half my memory 
it seems even running a lot of stuff at the same time. My swap after weeks 
of running is still 0%. (swappiness 10)


i understand how device drivers can be nasty (graphics cards locking up the 
pci bus, wifi chips hogging the kernel for milliseconds at a time or


Actually I think with wifi chips it is the bus that gets hogged.

worse...) but it seems that a) either kernel preemption and real-time 
scheduling is terribly buggy or hand-wavey, or b) we're feeding each other 
snake-oil in recommending to disable userspace things that is running without 
rt privs.


As you yourself can attest, it does make a difference. I would suggest 
that there are some kernel drivers that are optimized for throughput over 
latency that have not yet been accounted for. Or some other things that 
are in their own way time constrained. Network traffic comes to mind. 
Network traffic comes when it comes and can only be buffered in hardware 
so long before packets get lost. However, as I said, even full packets are 
relatively small. What is the biggest data chunk that gets written to 
disk? Has anyone gone through kernel drivers looking for atomic parts that 
could be shortened? Is there a setting for maximum data size of a disk 
write/read? It appears there are ways to throttle disk access speed on a 
per-proccess basis.


Another one that is puzzling is CPU speed changes (AKA OnDemand). These 
happen very fast and should not cause trouble, but they do. It seems to 
me, just by watching a cpu speed monitor, that xruns happen at the point 
the cpu speed goes down only. Perhaps there is some timing loop somewhere 
that gets expanded that should not. I would think any timing should be 
done by timers that are not cpu speed dependant.


Honestly, these are just thoughts off the top of my head. I don't know the 
kernel code well enough to say (Means I have not looked at it). I just 
know that by turning certain things off, I can get lower latency without 
xruns over a 24hour period. (even just sitting idle streaming zeros)


--
Len Ovens
www.ovenwerks.net
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Tim Goetze]

[Fokke de Jong]
>I’m processing 32 sample-blocks at 48KHz but roughly every 0,6
>seconds I get a large spike in cpu usage. This cannot possibly be
>explained by my algorithm, because the load should be pretty stable. 
>
>I am measuring cpu load by getting the time with
>clock_gettime(CLOCK_MONOTONIC_RAW, timespec*) at the beginning and
>end of each callback. When converted to a percentage my cpu load
>hovers somewhere between 40 an 50% most of the time, but more or less
>every 900 callbacks (0.8 seconds there is a spike of more than 100%.

Have you checked if your per-callback CPU time measurement and that
reported by getrusage(2) are in agreement?

Cheers, Tim___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Joakim Hernberg]

On Mon, 25 Jan 2016 12:53:57 +0100
Joakim Hernberg  wrote:

> On Mon, 25 Jan 2016 12:23:09 +0100
> Jörn Nettingsmeier  wrote:
> 
> > i understand how device drivers can be nasty (graphics cards locking
> > up the pci bus, wifi chips hogging the kernel for milliseconds at a
> > time or worse...) but it seems that a) either kernel preemption and
> > real-time scheduling is terribly buggy or hand-wavey, or b) we're
> > feeding each other snake-oil in recommending to disable userspace
> > things that is running without rt privs.
> > 
> > i'd love to be educated on this.  

2 additional observations, thermal events (cpu overheating) are bad and
will give xruns.  Also I've had problems keeping audio files in my home
dir when using traditional hdd (rust), better to use a different hdd,
or a ssd instead.  Changing i/o scheduler or class and priorities for
CFQ didn't seem to bring any relief.

-- 

   Joakim
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Joakim Hernberg]

On Mon, 25 Jan 2016 06:57:14 -0800 (PST)
Len Ovens  wrote:

> On Mon, 25 Jan 2016, Joakim Hernberg wrote:
> 
> > I suppose hyperthreading could be a potential pitfall, but
> > personally I see no problems with it with my audio workloads on my
> > i7.  
> 
> hyperthread is only a problem with jack latency under 64/2... even on
> an older single core P4. (at least in my testing)

I can't remember back to the P4 days ;)  But I thought I'd mention it
for completeness as it totally breaks the concept of SCHED_FIFO
threads.  Good to know that it appears to be a non problem.

-- 

   Joakim
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Jörn Nettingsmeier]

hi *!


sorry to hijack this thread, but: when enquiring about latency tuning, 
one frequently encounters hints like "disable cron", "disable indexing 
services", "disable this, disable that".


however, none of those alleged culprits run with real-time privileges or 
access driver or kernel code which does. so how can they be a problem 
(and disabling them part of the solution)? i'm asking because i've got 
my own anecdotal evidence that it *does* make a difference...


i understand how device drivers can be nasty (graphics cards locking up 
the pci bus, wifi chips hogging the kernel for milliseconds at a time or 
worse...) but it seems that a) either kernel preemption and real-time 
scheduling is terribly buggy or hand-wavey, or b) we're feeding each 
other snake-oil in recommending to disable userspace things that is 
running without rt privs.


i'd love to be educated on this.


best,


jörn




--
Jörn Nettingsmeier
Lortzingstr. 11, 45128 Essen, Tel. +49 177 7937487

Meister für Veranstaltungstechnik (Bühne/Studio)
Tonmeister VDT

http://stackingdwarves.net

___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Joakim Hernberg]

On Sun, 24 Jan 2016 15:03:08 +0100
Fokke de Jong  wrote:

> I am measuring cpu load by getting the time with
> clock_gettime(CLOCK_MONOTONIC_RAW, timespec*) at the beginning and
> end of each callback. When converted to a percentage my cpu load
> hovers somewhere between 40 an 50% most of the time, but more or less
> every 900 callbacks (0.8 seconds there is a spike of more than 100%.
> 
> I remember reading somewhere that realtime threads cannot run more
> than .95s every second. That would be very bad if it actually meant
> my threads are blocked  run for a period of 50ms straight…

You can disable the realtime runtime throttling if that is a problem,
but I'd doubt that would be the reason for your troubles, and it's
unlikely to cause problems unless you're really maxing out the CPUs.

Other reasons could range from hardware/kernel modules that aren't
playing nice with RT to wifi drivers and the NVIDIA module also comes to
mind.  Your system could also be suffering from SMI/NMI that runs out
of BIOS and completely preempts the kernel.

There are testing utilities in a package called rt-tests.  Try to run
cyclictest like this (sudo) "cyclictest -S -m -p98".  cyclictest
schedules threads to be run at a certain time, and can be used to get a
pretty good idea of kernel scheduling latencies.  It's the max that is
interesting to us, and hopefully yours will be under 100us, but if you
have peaks that go much higher that could be a reason for audio
dropouts.

Best to run it in the background for a while, and even better would be
to put load on the system.  The hackbench utility can be used for that.

Another reason could be cpu power management, though on a modern intel
processor that appears to be a thing of the past.

I suppose hyperthreading could be a potential pitfall, but personally I
see no problems with it with my audio workloads on my i7.

-- 

   Joakim
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev

Re: [LAD] cpu spikes

[Joakim Hernberg]

On Mon, 25 Jan 2016 12:23:09 +0100
Jörn Nettingsmeier  wrote:

> i understand how device drivers can be nasty (graphics cards locking
> up the pci bus, wifi chips hogging the kernel for milliseconds at a
> time or worse...) but it seems that a) either kernel preemption and
> real-time scheduling is terribly buggy or hand-wavey, or b) we're
> feeding each other snake-oil in recommending to disable userspace
> things that is running without rt privs.
> 
> i'd love to be educated on this.

My 2 cents is that it's snakeoil ;)

AFAIK, the important things are.

1. Use a properly configured realtime patched kernel.

2. Set a high priority of the soundcard interrupt, something like 97 is
a good value.  (If using a USB soundcard, set the priority of the
interrupt servicing the USB hub instead).

3. Run Jack with realtime and memlocking enabled and at a priority of
80.

4. Make sure that you don't have any hardware/drivers that play havoc
with your kernel scheduling.  some WIFI adapters, NVIDIA, etc comes to
mind.

5. Make sure that the system isn't suffering from SMI/NMIs which
preempt the kernel and can take a long time to execute.  This can be
done with hwlatdetect script in the rt-tests package.

6. Use cyclictest from rt-tests to confirm that there are no latency
spikes in how the kernel schedules threads.

Possibly hyperthreading, cpu power management, etc could cause
problems, and I don't have experience with all hardware out there, but
IME on modern Intel hardware this isn't a problem.

JACK2 also has a very nice profiling tool that can give a good idea
about what is going on with the soundcard interrupt, clients, etc.

Apart from that I think most things are snake oil, or ancient Internet
lore, of course ymmv ;)

-- 

   Joakim
___
Linux-audio-dev mailing list
Linux-audio-dev@lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev