Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Zach Brown
I'm finally back from my travel and conference hiatus.. you guys have been busy! :) On Feb 13, 2007, at 6:20 AM, Ingo Molnar wrote: I'm pleased to announce the first release of the "Syslet" kernel feature and kernel subsystem, which provides generic asynchrous system call support: htt

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Davide Libenzi
On Wed, 14 Feb 2007, Jeremy Fitzhardinge wrote: > Davide Libenzi wrote: > >> Would this work? > >> > > > > Hopefully the API will simplify enough so that emulation will becomes > > easier. > > > > The big question in my mind is how all this stuff interacts with > signals. Can a blocked

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Jeremy Fitzhardinge
Davide Libenzi wrote: >> Would this work? >> > > Hopefully the API will simplify enough so that emulation will becomes > easier. > The big question in my mind is how all this stuff interacts with signals. Can a blocked syscall atom be interrupted by a signal? If so, what thread does it

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Davide Libenzi
On Wed, 14 Feb 2007, Jeremy Fitzhardinge wrote: > Are there any special semantics that result from running the syslet > atoms in kernel mode? If I wanted to, could I write a syslet emulation > in userspace that's functionally identical to a kernel-based > implementation? (Obviously the performan

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Jeremy Fitzhardinge
Ingo Molnar wrote: > Syslets consist of 'syslet atoms', where each atom represents a single > system-call. These atoms can be chained to each other: serially, in > branches or in loops. The return value of an executed atom is checked > against the condition flags. So an atom can specify 'exit on

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Davide Libenzi
On Wed, 14 Feb 2007, Ingo Molnar wrote: > > * Evgeniy Polyakov <[EMAIL PROTECTED]> wrote: > > > Let me clarify what I meant. There is only limited number of threads, > > which are supposed to execute blocking context, so when all they are > > used, main one will block too - I asked about possi

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Linus Torvalds
On Wed, 14 Feb 2007, Pavel Machek wrote: > > Ouch, yet another interpretter in kernel :-(. Can we reuse acpi or > something? Hah. You make the joke! I get it! Mwahahahaa! Linus - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Pavel Machek
Hi! > The boring details: > > Syslets consist of 'syslet atoms', where each atom represents a single > system-call. These atoms can be chained to each other: serially, in > branches or in loops. The return value of an executed atom is checked > against the condition flags. So an atom can spe

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Evgeniy Polyakov
On Wed, Feb 14, 2007 at 11:37:31AM +0100, Ingo Molnar ([EMAIL PROTECTED]) wrote: > > Let me clarify what I meant. There is only limited number of threads, > > which are supposed to execute blocking context, so when all they are > > used, main one will block too - I asked about possibility to reus

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Alan
> > Ohh. OpenVMS lives forever ;) Me likeee ;) > > hm, i dont know OpenVMS - but googled around a bit for 'VMS > asynchronous' and it gave me this: VMS had SYS$QIO which is asynchronous I/O queueing with completions of sorts. You had to specifically remember if you wanted to a synchronous I/

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Ingo Molnar
* Evgeniy Polyakov <[EMAIL PROTECTED]> wrote: > Let me clarify what I meant. There is only limited number of threads, > which are supposed to execute blocking context, so when all they are > used, main one will block too - I asked about possibility to reuse the > same thread to execute queue o

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Evgeniy Polyakov
On Tue, Feb 13, 2007 at 11:18:10PM +0100, Ingo Molnar ([EMAIL PROTECTED]) wrote: > > * Evgeniy Polyakov <[EMAIL PROTECTED]> wrote: > > > [...] it still has a problem - syscall blocks and the same thread thus > > is not allowed to continue execution and fill the pipe - so what if > > system issu

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-14 Thread Ingo Molnar
* Davide Libenzi wrote: > > There's another problem AFAICS: > > > > - We woke up one of the cachemiss_loop threads in pick_new_async_thread > > > > - The threads wakes up, mark itself as busy, and look at the ->work > > pointer hoping to find something to work on > > > > But we never set t

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Davide Libenzi wrote: [...] > So the sys_async_exec task is going to block. Now, am I being really > tired, or the cachemiss fast return is simply not there? The former 8) pick_new_async_head() new_task->ah = ah; cachemiss_loop() for (;;) {

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Willy Tarreau
Hi Ingo ! On Tue, Feb 13, 2007 at 03:20:10PM +0100, Ingo Molnar wrote: > I'm pleased to announce the first release of the "Syslet" kernel feature > and kernel subsystem, which provides generic asynchrous system call > support: > >http://redhat.com/~mingo/syslet-patches/ > > Syslets are sma

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Ingo Molnar wrote: > I'm pleased to announce the first release of the "Syslet" kernel feature > and kernel subsystem, which provides generic asynchrous system call > support: > [...] Ok, I had little to time review the code, but it has been a long working day, so bear with

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Ingo Molnar wrote: > * Davide Libenzi wrote: > > > > If this is going to be a generic AIO subsystem: > > > > > > - Cancellation of peding request > > > > What about the busy_async_threads list becoming a hash/rb_tree indexed > > by syslet_atom ptr. A cancel would lookup t

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Ingo Molnar wrote: > > * Davide Libenzi wrote: > > > > Open issues: > > > If this is going to be a generic AIO subsystem: > > > > - Cancellation of pending request > > How about implementing aio_cancel() as a NOP. Can anyone prove that the > kernel didnt actually attemp

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ulrich Drepper
Ingo Molnar wrote: > really, what's the point behind aio_cancel()? - sequence aio_write() aio_cancel() aio_write() with both writes going to the same place must be predictably - think beyond files. Writes to sockets, ttys, they can block and cancel must abort them. Even for

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Olivier Galibert
On Tue, Feb 13, 2007 at 10:57:24PM +0100, Ingo Molnar wrote: > > * Davide Libenzi wrote: > > > > Open issues: > > > If this is going to be a generic AIO subsystem: > > > > - Cancellation of pending request > > How about implementing aio_cancel() as a NOP. Can anyone prove that the > kernel d

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Andi Kleen
> ok. The TID+signal approach i mentioned in the other reply should work. Not sure if a signal is good for this. It might conflict with existing strange historical semantics. > If it's frequent enough we could make this an explicit > sys_async_cancel(TID) API. Ideally there should be a new fun

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Andi Kleen <[EMAIL PROTECTED]> wrote: > > ok, that should work fine already - exit in the user context gets > > That would be a little heavy handed. I wouldn't expect my GUI program > to quit itself on cancel. And requiring it to create a new thread just > to exit on cancel would be also nas

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Dmitry Torokhov
Hi Ingo, On Tuesday 13 February 2007 15:39, Ingo Molnar wrote: > > * Dmitry Torokhov <[EMAIL PROTECTED]> wrote: > > > > What are the semantics of async sys_async_wait and async sys_async ? > > > > Ohh. OpenVMS lives forever ;) Me likeee ;) > > hm, i dont know OpenVMS - but googled around a

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Andi Kleen
On Tue, Feb 13, 2007 at 11:26:26PM +0100, Ingo Molnar wrote: > > * Andi Kleen <[EMAIL PROTECTED]> wrote: > > > > really, what's the point behind aio_cancel()? > > > > The main use case is when you open a file requester on a network file > > system where the server is down and you get tired of w

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Andi Kleen <[EMAIL PROTECTED]> wrote: > > really, what's the point behind aio_cancel()? > > The main use case is when you open a file requester on a network file > system where the server is down and you get tired of waiting and press > "Cancel" it should abort the hanging IO immediately. o

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Andi Kleen
Ingo Molnar <[EMAIL PROTECTED]> writes: > > really, what's the point behind aio_cancel()? The main use case is when you open a file requester on a network file system where the server is down and you get tired of waiting and press "Cancel" it should abort the hanging IO immediately. At least I w

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Evgeniy Polyakov <[EMAIL PROTECTED]> wrote: > > I have not received first mail with announcement yet, so I will place > > my thoughts here if you do not mind. > > An issue with sys_async_wait(): is is possible that events_left will > be setup too late so that all events are already ready and

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Evgeniy Polyakov <[EMAIL PROTECTED]> wrote: > [...] it still has a problem - syscall blocks and the same thread thus > is not allowed to continue execution and fill the pipe - so what if > system issues thousands of requests and there are only tens of working > thread at most. [...] the sam

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Davide Libenzi wrote: > > If this is going to be a generic AIO subsystem: > > > > - Cancellation of peding request > > What about the busy_async_threads list becoming a hash/rb_tree indexed > by syslet_atom ptr. A cancel would lookup the thread and send a signal > (of course, signal handli

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Davide Libenzi wrote: > > Open issues: > If this is going to be a generic AIO subsystem: > > - Cancellation of pending request How about implementing aio_cancel() as a NOP. Can anyone prove that the kernel didnt actually attempt to cancel that IO? [but unfortunately failed at doing so, be

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Davide Libenzi wrote: > If this is going to be a generic AIO subsystem: > > - Cancellation of peding request What about the busy_async_threads list becoming a hash/rb_tree indexed by syslet_atom ptr. A cancel would lookup the thread and send a signal (of course, signal han

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Dmitry Torokhov <[EMAIL PROTECTED]> wrote: > > What are the semantics of async sys_async_wait and async sys_async ? > > Ohh. OpenVMS lives forever ;) Me likeee ;) hm, i dont know OpenVMS - but googled around a bit for 'VMS asynchronous' and it gave me this: http://en.wikipedia.org/wik

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Benjamin LaHaise <[EMAIL PROTECTED]> wrote: > [...] interaction with set_fs()... hm, this one should already work in the current version, because addr_limit is in thread_info and hence stays with the async context. Or can you see any hole in it? Ingo - To unsubscribe from this list:

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Linus Torvalds wrote: > if (in_async_context()) > return -EINVAL; > > or similar. We need that "async_context()" function anyway for the other > cases where we can't do other things concurrently, like changing the UID. Yes, that's definitely better. Let'

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Davide Libenzi
On Tue, 13 Feb 2007, Ingo Molnar wrote: > As it might be obvious to some of you, the syslet subsystem takes many > ideas and experience from my Tux in-kernel webserver :) The syslet code > originates from a heavy rewrite of the Tux-atom and the Tux-cachemiss > infrastructure. > > Open issues:

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Evgeniy Polyakov
> I have not received first mail with announcement yet, so I will place > my thoughts here if you do not mind. An issue with sys_async_wait(): is is possible that events_left will be setup too late so that all events are already ready and thus sys_async_wait() can wait forever (or until next $sys

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Evgeniy Polyakov
On Tue, Feb 13, 2007 at 05:56:42PM +0100, Ingo Molnar ([EMAIL PROTECTED]) wrote: > > * Benjamin LaHaise <[EMAIL PROTECTED]> wrote: > > > > > Open issues: > > > > > > Let me add some more > > > > Also: FPU state (especially important with the FPU and SSE memory copy > > variants), segment regis

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Linus Torvalds <[EMAIL PROTECTED]> wrote: > Ie, we could just add to "do_fork()" (which is where all of the > vfork/clone/fork cases end up) a simple case like > > err = wait_async_context(); > if (err) > return err; > > or > > if (in_async_context()) >

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Benjamin LaHaise <[EMAIL PROTECTED]> wrote: > > > Open issues: > > > > Let me add some more > > Also: FPU state (especially important with the FPU and SSE memory copy > variants), segment register bases on x86-64, interaction with > set_fs()... agreed - i'll fix this. But i can see no big

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Andi Kleen <[EMAIL PROTECTED]> wrote: > > sys_exec and other security boundaries must be synchronous > > only and not allow async "spill over" (consider setuid async binary > > patching) > > He probably would need some generalization of Andrea's seccomp work. > Perhaps using bitmaps? Fo

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
* Alan <[EMAIL PROTECTED]> wrote: > > A syslet is executed opportunistically: i.e. the syslet subsystem > > assumes that the syslet will not block, and it will switch to a > > cachemiss kernel thread from the scheduler. This means that even a > > How is scheduler fairness maintained ? and what

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Linus Torvalds
On Tue, 13 Feb 2007, Andi Kleen wrote: > > sys_exec and other security boundaries must be synchronous only > > and not allow async "spill over" (consider setuid async binary patching) > > He probably would need some generalization of Andrea's seccomp work. > Perhaps using bitmaps? For paran

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread bert hubert
On Tue, Feb 13, 2007 at 09:58:48AM -0500, Benjamin LaHaise wrote: > not present is mandatory). I have looked into exactly this approach, and > it's only cheaper if the code is incomplete. Linux's native threads are > pretty damned good. Cheaper in time or in memory? Iow, would you be able to

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Dmitry Torokhov
On 2/13/07, Alan <[EMAIL PROTECTED]> wrote: > A syslet is executed opportunistically: i.e. the syslet subsystem > assumes that the syslet will not block, and it will switch to a > cachemiss kernel thread from the scheduler. This means that even a How is scheduler fairness maintained ? and what i

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Andi Kleen
Alan <[EMAIL PROTECTED]> writes: Funny, it sounds like batch() on stereoids @) Ok with an async context it becomes somewhat more interesting. > sys_setuid/gid/etc need to be synchronous only and not occur > while other async syscalls are running in parallel to meet current kernel > assump

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Arjan van de Ven
On Tue, 2007-02-13 at 09:58 -0500, Benjamin LaHaise wrote: > On Tue, Feb 13, 2007 at 03:00:19PM +, Alan wrote: > > > Open issues: > > > > Let me add some more > > Also: FPU state (especially important with the FPU and SSE memory copy > variants) are these preserved over explicit system call

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Benjamin LaHaise
On Tue, Feb 13, 2007 at 03:00:19PM +, Alan wrote: > > Open issues: > > Let me add some more Also: FPU state (especially important with the FPU and SSE memory copy variants), segment register bases on x86-64, interaction with set_fs()... There is no easy way of getting around the full threa

Re: [patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Alan
> A syslet is executed opportunistically: i.e. the syslet subsystem > assumes that the syslet will not block, and it will switch to a > cachemiss kernel thread from the scheduler. This means that even a How is scheduler fairness maintained ? and what is done for resource accounting here ? > th

[patch 00/11] ANNOUNCE: "Syslets", generic asynchronous system call support

2007-02-13 Thread Ingo Molnar
I'm pleased to announce the first release of the "Syslet" kernel feature and kernel subsystem, which provides generic asynchrous system call support: http://redhat.com/~mingo/syslet-patches/ Syslets are small, simple, lightweight programs (consisting of system-calls, 'atoms') that the kerne