RE: yield API

[David Schwartz]

Kyle Moffett wrote:

> That is a *terrible* disgusting way to use yield.  Better options:
>(1) inotify/dnotify

Sure, tie yourself to a Linux-specific mechanism that may or may not work
over things like NFS. That's much worse.

>(2) create a "foo.lock" file and put the mutex in that

Right, tie yourself to process-shared mutexes which historically weren't
available on Linux. That's much better than an option that's been stable for
a decade.

>(3) just start with the check-file-and-sleep loop.

How is that better? There is literally no improvement, since the first check
will (almost) always fail.

> > Now is this the best way to handle this situation? No.  Does it
> > work better than just doing the wait loop from the start? Yes.
>
> It works better than doing the wait-loop from the start?  What
> evidence do you provide to support this assertion?

The evidence is that more than half the time, this avoids the sleep. That
means it has zero cost, since the yield is no heavier than a sleep would be,
and has a possible benefit, since the first sleep may be too long.

> Specifically, in
> the first case you tell the kernel "I'm waiting for something but I
> don't know what it is or how long it will take"; while in the second
> case you tell the kernel "I'm waiting for something that will take
> exactly X milliseconds, even though I don't know what it is.  If you
> really want something similar to the old behavior then just replace
> the "sched_yield()" call with a proper sleep for the estimated time
> it will take the program to create the file.

The problem is that if the estimate is too short, pre-emption will result in
a huge performance drop. If the estimate is too long, there will be some
wasted CPU. What was the claimed benefit of doing this again?

> > Is this a good way to use sched_yield()? Maybe, maybe not.  But it
> > *is* an actual use of the API in a real app.

> We weren't looking for "actual uses", especially not in binary-only
> apps.  What we are looking for is optimal uses of sched_yield(); ones
> where that is the best alternative.  This... certainly isn't.

Your standards for "optimal" are totally unrealistic. In his case, it was
optimal. Using platform-specific optimizations would have meant more
development and test time for minimal benefit. Sleeping first would have had
some performance cost and no benefit. In his case, sched_yield was optimal.
Really.

DS


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[Kyle Moffett]

On Dec 12, 2007, at 17:39:15, Jesper Juhl wrote:

On 02/10/2007, Ingo Molnar <[EMAIL PROTECTED]> wrote:
sched_yield() has been around for a decade (about three times  
longer than futexes were around), so if it's useful, it sure  
should have grown some 'crown jewel' app that uses it and shows  
off its advantages, compared to other locking approaches, right?


I have one example of sched_yield() use in a real app.  
Unfortunately it's proprietary so I can't show you the source, but  
I can tell you how it's used.


The case is this:  Process A forks process B. Process B does some  
work that takes aproximately between 50 and 1000ms to complete  
(varies), then it creates a file and continues to do other work.   
Process A needs to wait for the file B creates before it can  
continue. Process A *could* immediately go into some kind of "check  
for file; sleep n ms" loop, but instead it starts off by calling  
sched_yield() to give process B a chance to run and hopefully get  
to the point where it has created the file before process A is  
again scheduled and starts to look for it - after the single sched  
yield call, process A does indeed go into a "check for file; sleep  
250ms;" loop, but most of the time the initial sched_yield() call  
actually results in the file being present without having to loop  
like that.


That is a *terrible* disgusting way to use yield.  Better options:
  (1) inotify/dnotify
  (2) create a "foo.lock" file and put the mutex in that
  (3) just start with the check-file-and-sleep loop.


Now is this the best way to handle this situation? No.  Does it  
work better than just doing the wait loop from the start? Yes.


It works better than doing the wait-loop from the start?  What  
evidence do you provide to support this assertion?  Specifically, in  
the first case you tell the kernel "I'm waiting for something but I  
don't know what it is or how long it will take"; while in the second  
case you tell the kernel "I'm waiting for something that will take  
exactly X milliseconds, even though I don't know what it is.  If you  
really want something similar to the old behavior then just replace  
the "sched_yield()" call with a proper sleep for the estimated time  
it will take the program to create the file.



Is this a good way to use sched_yield()? Maybe, maybe not.  But it  
*is* an actual use of the API in a real app.


We weren't looking for "actual uses", especially not in binary-only  
apps.  What we are looking for is optimal uses of sched_yield(); ones  
where that is the best alternative.  This... certainly isn't.


Cheers,
Kyle Moffett

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[Jesper Juhl]

On 02/10/2007, Ingo Molnar <[EMAIL PROTECTED]> wrote:
>
> * David Schwartz <[EMAIL PROTECTED]> wrote:
>
> > > These are generic statements, but i'm _really_ interested in the
> > > specifics. Real, specific code that i can look at. The typical Linux
> > > distro consists of in execess of 500 millions of lines of code, in
> > > tens of thousands of apps, so there really must be some good, valid
> > > and "right" use of sched_yield() somewhere in there, in some
> > > mainstream app, right? (because, as you might have guessed it, in
> > > the past decade of sched_yield() existence i _have_ seen my share of
> > > sched_yield() utilizing user-space code, and at the moment i'm not
> > > really impressed by those examples.)
> >
> > Maybe, maybe not. Even if so, it would be very difficult to find.
> > Simply grepping for sched_yield is not going to help because
> > determining whether a given use of sched_yield is smart is not going
> > to be easy.
>
> sched_yield() has been around for a decade (about three times longer
> than futexes were around), so if it's useful, it sure should have grown
> some 'crown jewel' app that uses it and shows off its advantages,
> compared to other locking approaches, right?
>

I have one example of sched_yield() use in a real app. Unfortunately
it's proprietary so I can't show you the source, but I can tell you
how it's used.

The case is this:  Process A forks process B. Process B does some work
that takes aproximately between 50 and 1000ms to complete (varies),
then it creates a file and continues to do other work.  Process A
needs to wait for the file B creates before it can continue.
Process A *could* immediately go into some kind of "check for file;
sleep n ms" loop, but instead it starts off by calling sched_yield()
to give process B a chance to run and hopefully get to the point where
it has created the file before process A is again scheduled and starts
to look for it - after the single sched yield call, process A does
indeed go into a "check for file; sleep 250ms;" loop, but most of the
time the initial sched_yield() call actually results in the file being
present without having to loop like that.

Now is this the best way to handle this situation? No.
Does it work better than just doing the wait loop from the start? Yes.
Is this a good way to use sched_yield()? Maybe, maybe not.  But it
*is* an actual use of the API in a real app.

> For example, if you asked me whether pipes are the best thing for
> certain apps, i could immediately show you tons of examples where they
> are. Same for sockets. Or RT priorities. Or nice levels. Or futexes. Or
> just about any other core kernel concept or API.

True. But in the app I'm talking about above, rewriting the code to
communicate over a pipe, socket or anything else would have been too
large a change to make (released product, can't risk introducing (new)
bugs).

>Your notion that
> showing a good example of an API would be "difficult" because it's hard
> to determine "smart" use is not tenable i believe and does not
> adequately refute my pretty plain-meaning "it does not exist" assertion.
>

I agree that sched_yield() is not a very good API. I also agree that
our use of it is not the best solution to the problem we wanted to
solve, but it actually works pretty well most of the time.

> If then this is one more supporting proof for the fundamental weakness
> of the sched_yield() API. Rarely are we able to so universally condemn
> an API: real-life is usually more varied and even for theoretically
> poorly defined APIs _some_ sort of legitimate use does grow up.
>
> APIs that are not in any real, meaningful use, despite a decade of
> presence are not really interesting to me personally. (especially in
> this case where we know exactly _why_ the API is used so rarely.) Sure
> we'll continue to support it in the best possible way, with the usual
> kernel maintainance policy: without hurting other, more commonly used
> APIs. That was the principle we followed in previous schedulers too. And
> if anyone has a patch to make sched_yield() better than it is today, i'm
> of course interested in it.
>
Just for the record; for our use, sched_yield() seems to work just
fine both with older and newer kernels, so from my point of view the
new scheduler is doing fine in this regard.

-- 
Jesper Juhl <[EMAIL PROTECTED]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please  http://www.expita.com/nomime.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[Eric St-Laurent]

On Tue, 2007-10-02 at 08:46 +0200, Ingo Molnar wrote:

[...]

> APIs that are not in any real, meaningful use, despite a decade of 
> presence are not really interesting to me personally. (especially in 
> this case where we know exactly _why_ the API is used so rarely.) Sure 
> we'll continue to support it in the best possible way, with the usual 
> kernel maintainance policy: without hurting other, more commonly used 
> APIs. That was the principle we followed in previous schedulers too. And 
> if anyone has a patch to make sched_yield() better than it is today, i'm 
> of course interested in it.

Do you still have intentions to add a directed yield API?  I remember
seeing it in the earlier CFS patches.


- Eric


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[Douglas McNaught]

"linux-os \(Dick Johnson\)" <[EMAIL PROTECTED]> writes:

> Whether or not there is a POSIX definition of sched_yield(),
> there is a need for something that will give up the CPU
> and not busy-wait. There are many control applications
> where state-machines are kept in user-mode code. The code
> waits for an event. It shouldn't be spinning, wasting
> CPU time, when the kernel can be doing file and network
> I/O with the wasted CPU cycles.

These "control applications" would be real-time processes, for which
(AIUI) sched_yield() behavior is completely well-defined and
implemented as such by Linux.  The question here is how useful the
call is for SCHED_OTHER (non-real-time) processes, for which it has no
well-defined semantics.

-Doug
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[linux-os (Dick Johnson)]

On Tue, 2 Oct 2007, Ingo Molnar wrote:

>
> * David Schwartz <[EMAIL PROTECTED]> wrote:
>
>>> These are generic statements, but i'm _really_ interested in the
>>> specifics. Real, specific code that i can look at. The typical Linux
>>> distro consists of in execess of 500 millions of lines of code, in
>>> tens of thousands of apps, so there really must be some good, valid
>>> and "right" use of sched_yield() somewhere in there, in some
>>> mainstream app, right? (because, as you might have guessed it, in
>>> the past decade of sched_yield() existence i _have_ seen my share of
>>> sched_yield() utilizing user-space code, and at the moment i'm not
>>> really impressed by those examples.)
>>
>> Maybe, maybe not. Even if so, it would be very difficult to find.
>> Simply grepping for sched_yield is not going to help because
>> determining whether a given use of sched_yield is smart is not going
>> to be easy.
>
> sched_yield() has been around for a decade (about three times longer
> than futexes were around), so if it's useful, it sure should have grown
> some 'crown jewel' app that uses it and shows off its advantages,
> compared to other locking approaches, right?
>
> For example, if you asked me whether pipes are the best thing for
> certain apps, i could immediately show you tons of examples where they
> are. Same for sockets. Or RT priorities. Or nice levels. Or futexes. Or
> just about any other core kernel concept or API. Your notion that
> showing a good example of an API would be "difficult" because it's hard
> to determine "smart" use is not tenable i believe and does not
> adequately refute my pretty plain-meaning "it does not exist" assertion.
>
> If then this is one more supporting proof for the fundamental weakness
> of the sched_yield() API. Rarely are we able to so universally condemn
> an API: real-life is usually more varied and even for theoretically
> poorly defined APIs _some_ sort of legitimate use does grow up.
>
> APIs that are not in any real, meaningful use, despite a decade of
> presence are not really interesting to me personally. (especially in
> this case where we know exactly _why_ the API is used so rarely.) Sure
> we'll continue to support it in the best possible way, with the usual
> kernel maintainance policy: without hurting other, more commonly used
> APIs. That was the principle we followed in previous schedulers too. And
> if anyone has a patch to make sched_yield() better than it is today, i'm
> of course interested in it.
>
>   Ingo

But sched_yield() on Linux never did what the majority of
programmers assumed it would do (give up the CPU to some
runnable processes for the rest of the time-slice). Instead,
it just appeared to spin in the kernel. Therefore, those
who needed a sched_yield(), just used usleep().

Whether or not there is a POSIX definition of sched_yield(),
there is a need for something that will give up the CPU
and not busy-wait. There are many control applications
where state-machines are kept in user-mode code. The code
waits for an event. It shouldn't be spinning, wasting
CPU time, when the kernel can be doing file and network
I/O with the wasted CPU cycles.

So, just because sched_yield() doesn't work as expected,
is not the reason to get rid of it.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.16.24 on an i686 machine (5592.59 BogoMips).
My book : http://www.AbominableFirebug.com/
_



The information transmitted in this message is confidential and may be 
privileged.  Any review, retransmission, dissemination, or other use of this 
information by persons or entities other than the intended recipient is 
prohibited.  If you are not the intended recipient, please notify Analogic 
Corporation immediately - by replying to this message or by sending an email to 
[EMAIL PROTECTED] - and destroy all copies of this information, including any 
attachments, without reading or disclosing them.

Thank you.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: yield API

[Ingo Molnar]

* David Schwartz <[EMAIL PROTECTED]> wrote:

> > These are generic statements, but i'm _really_ interested in the 
> > specifics. Real, specific code that i can look at. The typical Linux 
> > distro consists of in execess of 500 millions of lines of code, in 
> > tens of thousands of apps, so there really must be some good, valid 
> > and "right" use of sched_yield() somewhere in there, in some 
> > mainstream app, right? (because, as you might have guessed it, in 
> > the past decade of sched_yield() existence i _have_ seen my share of 
> > sched_yield() utilizing user-space code, and at the moment i'm not 
> > really impressed by those examples.)
> 
> Maybe, maybe not. Even if so, it would be very difficult to find. 
> Simply grepping for sched_yield is not going to help because 
> determining whether a given use of sched_yield is smart is not going 
> to be easy.

sched_yield() has been around for a decade (about three times longer 
than futexes were around), so if it's useful, it sure should have grown 
some 'crown jewel' app that uses it and shows off its advantages, 
compared to other locking approaches, right?

For example, if you asked me whether pipes are the best thing for 
certain apps, i could immediately show you tons of examples where they 
are. Same for sockets. Or RT priorities. Or nice levels. Or futexes. Or 
just about any other core kernel concept or API. Your notion that 
showing a good example of an API would be "difficult" because it's hard 
to determine "smart" use is not tenable i believe and does not 
adequately refute my pretty plain-meaning "it does not exist" assertion.

If then this is one more supporting proof for the fundamental weakness 
of the sched_yield() API. Rarely are we able to so universally condemn 
an API: real-life is usually more varied and even for theoretically 
poorly defined APIs _some_ sort of legitimate use does grow up.

APIs that are not in any real, meaningful use, despite a decade of 
presence are not really interesting to me personally. (especially in 
this case where we know exactly _why_ the API is used so rarely.) Sure 
we'll continue to support it in the best possible way, with the usual 
kernel maintainance policy: without hurting other, more commonly used 
APIs. That was the principle we followed in previous schedulers too. And 
if anyone has a patch to make sched_yield() better than it is today, i'm 
of course interested in it.

Ingo
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [EMAIL PROTECTED]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/