Re: Startvation of realtime piority threads
Thanks. I'll try to back port locally. Sushanth --- On Tue, 4/10/12, John Baldwin wrote: > From: John Baldwin > Subject: Re: Startvation of realtime piority threads > To: "Sushanth Rai" > Cc: freebsd-hackers@freebsd.org > Date: Tuesday, April 10, 2012, 6:57 AM > On Monday, April 09, 2012 4:32:24 pm > Sushanth Rai wrote: > > I'm using stock 7.2. The priorities as defined in > priority.h are in this range: > > > > /* > > * Priorities range from 0 to 255, but differences > of less then 4 (RQ_PPQ) > > * are insignificant. Ranges are as > follows: > > * > > * Interrupt threads: > 0 - 63 > > * Top half kernel threads: > 64 - 127 > > * Realtime user threads: > 128 - 159 > > * Time sharing user threads: 160 > - 223 > > * Idle user threads: > 224 - 255 > > * > > * XXX If/When the specific interrupt thread and > top half thread ranges > > * disappear, a larger range can be used for user > processes. > > */ > > > > The trouble is with vm_waitpfault(), which explicitly > sleeps at PUSER. > > Ah, yes, PUSER is the one Pxxx not in "top half kernel > threads". You can patch > that locally, but you may have better lucking using 9.0 (or > backporting my > fixes in 9.0 back to 7 or 8). They were too invasive > to backport to FreeBSD > 7/8, but you could still do it locally (I've used them at > work on both 7 and 8). > > -- > John Baldwin > ___ > freebsd-hackers@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On Monday, April 09, 2012 4:32:24 pm Sushanth Rai wrote: > I'm using stock 7.2. The priorities as defined in priority.h are in this > range: > > /* > * Priorities range from 0 to 255, but differences of less then 4 (RQ_PPQ) > * are insignificant. Ranges are as follows: > * > * Interrupt threads: 0 - 63 > * Top half kernel threads: 64 - 127 > * Realtime user threads: 128 - 159 > * Time sharing user threads: 160 - 223 > * Idle user threads: 224 - 255 > * > * XXX If/When the specific interrupt thread and top half thread ranges > * disappear, a larger range can be used for user processes. > */ > > The trouble is with vm_waitpfault(), which explicitly sleeps at PUSER. Ah, yes, PUSER is the one Pxxx not in "top half kernel threads". You can patch that locally, but you may have better lucking using 9.0 (or backporting my fixes in 9.0 back to 7 or 8). They were too invasive to backport to FreeBSD 7/8, but you could still do it locally (I've used them at work on both 7 and 8). -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
I'm using stock 7.2. The priorities as defined in priority.h are in this range: /* * Priorities range from 0 to 255, but differences of less then 4 (RQ_PPQ) * are insignificant. Ranges are as follows: * * Interrupt threads: 0 - 63 * Top half kernel threads: 64 - 127 * Realtime user threads: 128 - 159 * Time sharing user threads: 160 - 223 * Idle user threads: 224 - 255 * * XXX If/When the specific interrupt thread and top half thread ranges * disappear, a larger range can be used for user processes. */ The trouble is with vm_waitpfault(), which explicitly sleeps at PUSER. Sushanth --- On Mon, 4/9/12, John Baldwin wrote: > From: John Baldwin > Subject: Re: Startvation of realtime piority threads > To: "Sushanth Rai" > Cc: freebsd-hackers@freebsd.org > Date: Monday, April 9, 2012, 11:37 AM > On Monday, April 09, 2012 2:08:50 pm > Sushanth Rai wrote: > > I'm on 7.2. sched_sleep() on 7.2 just records the sleep > time. That's why I > though _sleep might the right place to do the check. > > Nah, sched_sleep() is more accurate since the sleep priority > can have other > side effects. > > Hmm, in stock 7.2, the rtprio range is below things like > PVM, etc., so that > shouldn't actually be buggy in that regard. I fixed > this in 9.0 and HEAD > when I moved the rtprio range up above the kernel sleep > priorities. Are > you using local patches to 7.2 to raise the priority of > rtprio threads? > > > Thanks, > > Sushanth > > > > --- On Mon, 4/9/12, John Baldwin > wrote: > > > > > From: John Baldwin > > > Subject: Re: Startvation of realtime piority > threads > > > To: "Sushanth Rai" > > > Cc: freebsd-hackers@freebsd.org > > > Date: Monday, April 9, 2012, 9:17 AM > > > On Thursday, April 05, 2012 9:08:24 > > > pm Sushanth Rai wrote: > > > > I understand the downside of badly written > realtime > > > app. In my case > > > application runs in userspace without making much > syscalls > > > and by all means it > > > is a well behaved application. Yes, I can wire > memory, > > > change the application > > > to use mutex instead of spinlock and those changes > should > > > help but they are > > > still working around the problem. I still believe > kernel > > > should not lower the > > > realtime priority when blocking on resources. This > can lead > > > to priority > > > inversion, especially since these threads run at > fixed > > > priorities and kernel > > > doesn't muck with them. > > > > > > > > As you suggested _sleep() should not adjust > the > > > priorities for realtime > > > threads. > > > > > > Hmm, sched_sleep() for both SCHED_4BSD and > SCHED_ULE already > > > does the right > > > thing here in HEAD. > > > > > > if > (PRI_BASE(td->td_pri_class) != > > > PRI_TIMESHARE) > > > return; > > > > > > Which OS version did you see this on? > > > > > > -- > > > John Baldwin > > > > > > > -- > John Baldwin > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On Monday, April 09, 2012 2:08:50 pm Sushanth Rai wrote: > I'm on 7.2. sched_sleep() on 7.2 just records the sleep time. That's why I though _sleep might the right place to do the check. Nah, sched_sleep() is more accurate since the sleep priority can have other side effects. Hmm, in stock 7.2, the rtprio range is below things like PVM, etc., so that shouldn't actually be buggy in that regard. I fixed this in 9.0 and HEAD when I moved the rtprio range up above the kernel sleep priorities. Are you using local patches to 7.2 to raise the priority of rtprio threads? > Thanks, > Sushanth > > --- On Mon, 4/9/12, John Baldwin wrote: > > > From: John Baldwin > > Subject: Re: Startvation of realtime piority threads > > To: "Sushanth Rai" > > Cc: freebsd-hackers@freebsd.org > > Date: Monday, April 9, 2012, 9:17 AM > > On Thursday, April 05, 2012 9:08:24 > > pm Sushanth Rai wrote: > > > I understand the downside of badly written realtime > > app. In my case > > application runs in userspace without making much syscalls > > and by all means it > > is a well behaved application. Yes, I can wire memory, > > change the application > > to use mutex instead of spinlock and those changes should > > help but they are > > still working around the problem. I still believe kernel > > should not lower the > > realtime priority when blocking on resources. This can lead > > to priority > > inversion, especially since these threads run at fixed > > priorities and kernel > > doesn't muck with them. > > > > > > As you suggested _sleep() should not adjust the > > priorities for realtime > > threads. > > > > Hmm, sched_sleep() for both SCHED_4BSD and SCHED_ULE already > > does the right > > thing here in HEAD. > > > > if (PRI_BASE(td->td_pri_class) != > > PRI_TIMESHARE) > > return; > > > > Which OS version did you see this on? > > > > -- > > John Baldwin > > > -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
I'm on 7.2. sched_sleep() on 7.2 just records the sleep time. That's why I though _sleep might the right place to do the check. Thanks, Sushanth --- On Mon, 4/9/12, John Baldwin wrote: > From: John Baldwin > Subject: Re: Startvation of realtime piority threads > To: "Sushanth Rai" > Cc: freebsd-hackers@freebsd.org > Date: Monday, April 9, 2012, 9:17 AM > On Thursday, April 05, 2012 9:08:24 > pm Sushanth Rai wrote: > > I understand the downside of badly written realtime > app. In my case > application runs in userspace without making much syscalls > and by all means it > is a well behaved application. Yes, I can wire memory, > change the application > to use mutex instead of spinlock and those changes should > help but they are > still working around the problem. I still believe kernel > should not lower the > realtime priority when blocking on resources. This can lead > to priority > inversion, especially since these threads run at fixed > priorities and kernel > doesn't muck with them. > > > > As you suggested _sleep() should not adjust the > priorities for realtime > threads. > > Hmm, sched_sleep() for both SCHED_4BSD and SCHED_ULE already > does the right > thing here in HEAD. > > if (PRI_BASE(td->td_pri_class) != > PRI_TIMESHARE) > return; > > Which OS version did you see this on? > > -- > John Baldwin > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On Thursday, April 05, 2012 9:08:24 pm Sushanth Rai wrote: > I understand the downside of badly written realtime app. In my case application runs in userspace without making much syscalls and by all means it is a well behaved application. Yes, I can wire memory, change the application to use mutex instead of spinlock and those changes should help but they are still working around the problem. I still believe kernel should not lower the realtime priority when blocking on resources. This can lead to priority inversion, especially since these threads run at fixed priorities and kernel doesn't muck with them. > > As you suggested _sleep() should not adjust the priorities for realtime threads. Hmm, sched_sleep() for both SCHED_4BSD and SCHED_ULE already does the right thing here in HEAD. if (PRI_BASE(td->td_pri_class) != PRI_TIMESHARE) return; Which OS version did you see this on? -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
I understand the downside of badly written realtime app. In my case application runs in userspace without making much syscalls and by all means it is a well behaved application. Yes, I can wire memory, change the application to use mutex instead of spinlock and those changes should help but they are still working around the problem. I still believe kernel should not lower the realtime priority when blocking on resources. This can lead to priority inversion, especially since these threads run at fixed priorities and kernel doesn't muck with them. As you suggested _sleep() should not adjust the priorities for realtime threads. Thanks, Sushanth --- On Thu, 4/5/12, John Baldwin wrote: > From: John Baldwin > Subject: Re: Startvation of realtime piority threads > To: freebsd-hackers@freebsd.org, davi...@freebsd.org > Date: Thursday, April 5, 2012, 9:01 AM > On Thursday, April 05, 2012 1:07:55 > am David Xu wrote: > > On 2012/4/5 11:56, Konstantin Belousov wrote: > > > On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth > Rai wrote: > > >> I have a multithreaded user space program that > basically runs at realtime > priority. Synchronization between threads are done using > spinlock. When > running this program on a SMP system under heavy memory > pressure I see that > thread holding the spinlock is starved out of cpu. The cpus > are effectively > consumed by other threads that are spinning for lock to > become available. > > >> > > >> After instrumenting the kernel a little bit > what I found was that under > memory pressure, when the user thread holding the spinlock > traps into the > kernel due to page fault, that thread sleeps until the free > pages are > available. The thread sleeps PUSER priority (within > vm_waitpfault()). When it > is ready to run, it is queued at PUSER priority even thought > it's base > priority is realtime. The other siblings threads that are > spinning at realtime > priority to acquire the spinlock starves the owner of > spinlock. > > >> > > >> I was wondering if the sleep in > vm_waitpfault() should be a > MAX(td_user_pri, PUSER) instead of just PUSER. I'm running > on 7.2 and it looks > like this logic is the same in the trunk. > > > It just so happen that your program stumbles upon > a single sleep point in > > > the kernel. If for whatever reason the thread in > kernel is put off CPU > > > due to failure to acquire any resource without > priority propagation, > > > you would get the same effect. Only blockable > primitives do priority > > > propagation, that are mutexes and rwlocks, AFAIR. > In other words, any > > > sx/lockmgr/sleep points are vulnerable to the same > issue. > > This is why I suggested that POSIX realtime priority > should not be > > boosted, it should be > > only higher than PRI_MIN_TIMESHARE but lower than any > priority all > > msleep() callers > > provided. The problem is userland realtime thread > 's busy looping code > > can cause > > starvation a thread in kernel which holding a critical > resource. > > In kernel we can avoid to write dead-loop code, but > userland code is not > > trustable. > > Note that you have to be root to be rtprio, and that there > is trustable > userland code (just because you haven't used any doesn't > mean it doesn't > exist). > > > If you search "Realtime thread priorities" in > 2010-december within @arch > > list. > > you may find the argument. > > I think the bug here is that sched_sleep() should not lower > the priority of > an rtprio process. It should arguably not raise the > priority of an idprio > process either, but sched_sleep() should probably only apply > to timesharing > threads. > > All that said, userland rtprio code is going to have to be > careful. It should > be using things like wired memory as Kostik suggested, and > probably avoiding > most system calls. You can definitely blow your foot > off quite easily in lots > of ways with rtprio. > > -- > John Baldwin > ___ > freebsd-hackers@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On Thursday, April 05, 2012 1:07:55 am David Xu wrote: > On 2012/4/5 11:56, Konstantin Belousov wrote: > > On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth Rai wrote: > >> I have a multithreaded user space program that basically runs at realtime priority. Synchronization between threads are done using spinlock. When running this program on a SMP system under heavy memory pressure I see that thread holding the spinlock is starved out of cpu. The cpus are effectively consumed by other threads that are spinning for lock to become available. > >> > >> After instrumenting the kernel a little bit what I found was that under memory pressure, when the user thread holding the spinlock traps into the kernel due to page fault, that thread sleeps until the free pages are available. The thread sleeps PUSER priority (within vm_waitpfault()). When it is ready to run, it is queued at PUSER priority even thought it's base priority is realtime. The other siblings threads that are spinning at realtime priority to acquire the spinlock starves the owner of spinlock. > >> > >> I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic is the same in the trunk. > > It just so happen that your program stumbles upon a single sleep point in > > the kernel. If for whatever reason the thread in kernel is put off CPU > > due to failure to acquire any resource without priority propagation, > > you would get the same effect. Only blockable primitives do priority > > propagation, that are mutexes and rwlocks, AFAIR. In other words, any > > sx/lockmgr/sleep points are vulnerable to the same issue. > This is why I suggested that POSIX realtime priority should not be > boosted, it should be > only higher than PRI_MIN_TIMESHARE but lower than any priority all > msleep() callers > provided. The problem is userland realtime thread 's busy looping code > can cause > starvation a thread in kernel which holding a critical resource. > In kernel we can avoid to write dead-loop code, but userland code is not > trustable. Note that you have to be root to be rtprio, and that there is trustable userland code (just because you haven't used any doesn't mean it doesn't exist). > If you search "Realtime thread priorities" in 2010-december within @arch > list. > you may find the argument. I think the bug here is that sched_sleep() should not lower the priority of an rtprio process. It should arguably not raise the priority of an idprio process either, but sched_sleep() should probably only apply to timesharing threads. All that said, userland rtprio code is going to have to be careful. It should be using things like wired memory as Kostik suggested, and probably avoiding most system calls. You can definitely blow your foot off quite easily in lots of ways with rtprio. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On 2012/4/5 11:56, Konstantin Belousov wrote: On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth Rai wrote: I have a multithreaded user space program that basically runs at realtime priority. Synchronization between threads are done using spinlock. When running this program on a SMP system under heavy memory pressure I see that thread holding the spinlock is starved out of cpu. The cpus are effectively consumed by other threads that are spinning for lock to become available. After instrumenting the kernel a little bit what I found was that under memory pressure, when the user thread holding the spinlock traps into the kernel due to page fault, that thread sleeps until the free pages are available. The thread sleeps PUSER priority (within vm_waitpfault()). When it is ready to run, it is queued at PUSER priority even thought it's base priority is realtime. The other siblings threads that are spinning at realtime priority to acquire the spinlock starves the owner of spinlock. I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic is the same in the trunk. It just so happen that your program stumbles upon a single sleep point in the kernel. If for whatever reason the thread in kernel is put off CPU due to failure to acquire any resource without priority propagation, you would get the same effect. Only blockable primitives do priority propagation, that are mutexes and rwlocks, AFAIR. In other words, any sx/lockmgr/sleep points are vulnerable to the same issue. This is why I suggested that POSIX realtime priority should not be boosted, it should be only higher than PRI_MIN_TIMESHARE but lower than any priority all msleep() callers provided. The problem is userland realtime thread 's busy looping code can cause starvation a thread in kernel which holding a critical resource. In kernel we can avoid to write dead-loop code, but userland code is not trustable. If you search "Realtime thread priorities" in 2010-december within @arch list. you may find the argument. Speaking of exactly your problem, did you considered wiring the memory of your realtime process ? This is a common practice, used e.g. by ntpd. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On 2012/4/5 9:54, Sushanth Rai wrote: I have a multithreaded user space program that basically runs at realtime priority. Synchronization between threads are done using spinlock. When running this program on a SMP system under heavy memory pressure I see that thread holding the spinlock is starved out of cpu. The cpus are effectively consumed by other threads that are spinning for lock to become available. After instrumenting the kernel a little bit what I found was that under memory pressure, when the user thread holding the spinlock traps into the kernel due to page fault, that thread sleeps until the free pages are available. The thread sleeps PUSER priority (within vm_waitpfault()). When it is ready to run, it is queued at PUSER priority even thought it's base priority is realtime. The other siblings threads that are spinning at realtime priority to acquire the spinlock starves the owner of spinlock. I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic is the same in the trunk. Thanks, Sushanth I think 7.2 still has libkse which supports static priority scheduling, if performance is not important but correctness, you may try libkse with process-scope threads, and use priority-inherit mutex to do locking. Kernel is known to be vulnerable to support user realtime threads. I think not every-locking primitive can support priority propagation, this is an issue. In userland, internal library mutexes are not priority-inherit, so starvation may happen too. If you know what you are doing, don't call such functions which uses internal mutexes, but this is rather difficult. Regards, David Xu ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Startvation of realtime piority threads
On Wed, Apr 04, 2012 at 06:54:06PM -0700, Sushanth Rai wrote: > I have a multithreaded user space program that basically runs at realtime > priority. Synchronization between threads are done using spinlock. When > running this program on a SMP system under heavy memory pressure I see that > thread holding the spinlock is starved out of cpu. The cpus are effectively > consumed by other threads that are spinning for lock to become available. > > After instrumenting the kernel a little bit what I found was that under > memory pressure, when the user thread holding the spinlock traps into the > kernel due to page fault, that thread sleeps until the free pages are > available. The thread sleeps PUSER priority (within vm_waitpfault()). When it > is ready to run, it is queued at PUSER priority even thought it's base > priority is realtime. The other siblings threads that are spinning at > realtime priority to acquire the spinlock starves the owner of spinlock. > > I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, > PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic > is the same in the trunk. It just so happen that your program stumbles upon a single sleep point in the kernel. If for whatever reason the thread in kernel is put off CPU due to failure to acquire any resource without priority propagation, you would get the same effect. Only blockable primitives do priority propagation, that are mutexes and rwlocks, AFAIR. In other words, any sx/lockmgr/sleep points are vulnerable to the same issue. Speaking of exactly your problem, did you considered wiring the memory of your realtime process ? This is a common practice, used e.g. by ntpd. pgpH0WHCLjZxr.pgp Description: PGP signature
Startvation of realtime piority threads
I have a multithreaded user space program that basically runs at realtime priority. Synchronization between threads are done using spinlock. When running this program on a SMP system under heavy memory pressure I see that thread holding the spinlock is starved out of cpu. The cpus are effectively consumed by other threads that are spinning for lock to become available. After instrumenting the kernel a little bit what I found was that under memory pressure, when the user thread holding the spinlock traps into the kernel due to page fault, that thread sleeps until the free pages are available. The thread sleeps PUSER priority (within vm_waitpfault()). When it is ready to run, it is queued at PUSER priority even thought it's base priority is realtime. The other siblings threads that are spinning at realtime priority to acquire the spinlock starves the owner of spinlock. I was wondering if the sleep in vm_waitpfault() should be a MAX(td_user_pri, PUSER) instead of just PUSER. I'm running on 7.2 and it looks like this logic is the same in the trunk. Thanks, Sushanth ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
