Re: [9fans] Quick question on stopping a process that waits for IO
erik quanstrom wrote: How? If there's a stop message already written to /proc/n/ctl. Once that is done, the process is guaranteed to be in 2 states and those states only: continue waiting for the I/O, being actually Stopped. Both of the don't let the scheduler take it to the runqueue. here's the senerio, i think (works fine on a single processor) a b acquire debug lock sleep complete io sched run a bit syscall wakeup But how "run a bit" could possibly happen if after the "stop" message being sent right after the "complete io" the "b" process goes into a "Stopped" state? Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
> >> The target process is *already* waiting for the IO stuck inside the > >> kernel. It is not on a runqueue, not it is considered to be places > >> there. > > > > since procwrite doesn't acquire anything other than the debug lock, > > how do you know? the proc could start up again before you notice. > > How? If there's a stop message already written to /proc/n/ctl. Once > that is done, the process is guaranteed to be in 2 states and those > states only: continue waiting for the I/O, being actually Stopped. > Both of the don't let the scheduler take it to the runqueue. here's the senerio, i think (works fine on a single processor) a b acquire debug lock sleep complete io sched run a bit syscall wakeup - erik
Re: [9fans] Quick question on stopping a process that waits for IO
> >> Did I miss something obvious? > > > > And this would be a million dollar question here. I don't > > think you did (although Eric (sic) constantly warns us of > > dragons), but on the other hand I have very little > > experience with the kernel itself. > > I hope somebody comments on the fencing that is or isn't needed here. Since > we have parts of the kernel peeking witout locks at ->procctl, I worry about > races, and wonder how, or if, the current design avoids them. memory fencing only forces the instruction stream to be read, write or r/w coherent with the local processor. it is of no help at all for concurrent access. supposing there is a problem, memory fencing will not help. you would need an ilock. i didn't spend enough time looking at pc/trap.c to convince myself one way or the other, but the main access (other than syscall tracing) seemed to be a read of the variable to transition to stopped state. this race is likely benign. you'll just go around again and catch it later. notes are hard. and the problem they're intended to solve is how to interrupt a single-threaded process. if you're writing something new and you think you might want to extend notes, i think you would be much happier using the thread library and living in user space. > in local store -- that drove the design of the Coyotos kernel to being a > strictly transactional system. AFAIR there it is always possible to tell a it's easy to have a design where you let the dragons play and you need to bring out some serious weaponry to deal with your problems. (and it is possible to get carried away looking for concurrency problems.) a good design like plan 9 keeps the dragons locked up, mostly. - erik
Re: [9fans] Quick question on stopping a process that waits for IO
On Thu, Nov 06, 2008 at 09:18:47PM -0800, Roman Shaposhnik wrote: > On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote: >> Would this suffice? > > It sounds like exactly the kind of thing I was talking about. OK. To reiterate what I said earlier, these kinds of "soonstop"-ed processes may still make some amount of progress in the kernel before becoming stopped, but they will indeed stop at their next scheduling opportunity, and ideally well before (when they go to return to userland). One could imagine a parallel "soonnote" which considered the note delivered even if the process was asleep in I/O. This might be the more appropriate thing for sending kill requests from the shell? Having not looked at the shell, this might be a bad idea; I'm not sure. >> Did I miss something obvious? > > And this would be a million dollar question here. I don't > think you did (although Eric constantly warns us of > dragons), but on the other hand I have very little > experience with the kernel itself. I hope somebody comments on the fencing that is or isn't needed here. Since we have parts of the kernel peeking witout locks at ->procctl, I worry about races, and wonder how, or if, the current design avoids them. ( Incidentally, if you're curious about in-kernel development and want to look at a strange system, it is exactly the species of dragons we have encountered here -- where the kernel has hijacked a userland thread to do driver operations, and is potentially sleeping with driver resources held or in local store -- that drove the design of the Coyotos kernel to being a strictly transactional system. AFAIR there it is always possible to tell a thread to kill itself after its current transaction or, if asleep or in commit phase, to force it out of stall queues and then kill it. ) --nwf; pgp81iSoj5Hin.pgp Description: PGP signature
Re: [9fans] Quick question on stopping a process that waits for IO
On Nov 4, 2008, at 4:34 AM, erik quanstrom wrote: So the question remains -- what is the proper way of putting a process that waits for an IO into a Stopped state? i don't think it's possible without changing the kernel. but it's a good question, why does it work this way? obviously one doesn't want to stop a process and kernel work in the middle of i/o. there may be locks involved, etc. but one could easily check for Proc_stopme on syscall exit. does anyone remember the thinking that went into this? I was waiting for somebody to chime in, but I guess no takers :-( Well, one last thing to report before we put this thread to rest would be the fact that acid(1) seems to be suffering from the same issue. If the process is doing I/O I can't modify the variables. Not a huge limitation, but a surprising one nonetheless if one draws from experience of working with conventional debuggers. Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
On Nov 5, 2008, at 4:55 AM, erik quanstrom wrote: I'm asking is -- "dear kernel, please don't advance this process even if you otherwise can". All I need is a frozen state so that I can not so easy on a multiprocessor. (unless you turn all but one processor off.) Hm. May be its getting late, but I can't quite see why that would be the case (or may be I didn't quite communicate the intent in my plea to the kernel ;-)) The target process is *already* waiting for the IO stuck inside the kernel. It is not on a runqueue, not it is considered to be places there. since procwrite doesn't acquire anything other than the debug lock, how do you know? the proc could start up again before you notice. How? If there's a stop message already written to /proc/n/ctl. Once that is done, the process is guaranteed to be in 2 states and those states only: continue waiting for the I/O, being actually Stopped. Both of the don't let the scheduler take it to the runqueue. the place to catch processes is on syscall entry and/or exit. stop messages are only checked on syscall entry. i have a feeling that you want something other than notes for your problem. If by the problem you mean stopping a process doing I/O than, I would gladly accept any solution be it notes or otherwise. cpu arranges that del works. Well, it doesn't on Stopped processes. Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote: Would this suffice? It sounds like exactly the kind of thing I was talking about. Did I miss something obvious? And this would be a million dollar question here. I don't think you did (although Eric constantly warns us of dragons), but on the other hand I have very little experience with the kernel itself. Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
On Sun, Nov 02, 2008 at 09:55:16PM -0800, Roman Shaposhnik wrote:
> Guys,
>
> when somebody tries to stop a process that is waiting for the IO the
> process
> doesn't get transferred to a Stopped state immediately but only when
> the scheduler sees it for the first time. This leads to a process
> writing to
> a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
>
> Is there any way I can poke the target process so that it gets attention
> from the scheduler an can be put in a Stopped state?
>
> Thanks,
> Roman.
Thinking aloud, and looking mostly at the port/ and pc/ code:
port/devproc.c:/^procstopwait will be called when we write "stop" into
/proc/n/ctl. Modulo details, this function will:
qlock p->debug
mark us as p's debugger
set p->procctl to Proc_stopme
drop p->debug
wait until p->state == Stopped by using procstopped()
return
If a process is stopped on I/O, that wait is going to take "a while" (until
I/O completes). Eventually, however, it will return into
pc/trap.c:/^syscall and will eventually run the code:
> if(scallnr!=RFORK && (up->procctl || up->nnote)){
> splhi();
> notify(ureg);
> }
whereupon we expect [1] that it sees that up->procctl is nonzero and the
process will put itself to sleep inside notify(), awake the waiter above,
and everything will be fine.
So it seems like it wouldn't be very hard to define a "soonstop" command
which considered processes blocked on I/O "as good as stopped" since there
are two outcomes:
After we set up->procctl, the target process returned to userland and will
sleep on the next trap they take or syscall they make. In this case, we
should set ourselves as p's debugger and wait as we have a bounded wait:
exactly one time quantum at most.
After we set up->procctl, the process is still blocked (or blocked again)
on I/O. They will not make it back to userland after this I/O completes.
In this case, we do not need to register ourselves as p's debugger and
may immediately return.
Subject to appropriate fencing (if required, I'm not sure; sim. [1]) it
seems that we can distinguish these cases in the requesting process.
User memory and registers (probably just the return register, tho') would be
subject to change by the kernel during the period where the process is
asleep, but this is not too dissimilar from the state of the world when
shared memory is in flight.
Would this suffice? Did I miss something obvious?
--nwf;
[1] It may be necessary to insert a memory fence after the setting of
p->procctl or before the read of up->procctl, since several places
do not acquire up->debug before reading up->procctl. Can somebody
comment?
pgpFEoVHZMQkk.pgp
Description: PGP signature
Re: [9fans] Quick question on stopping a process that waits for IO
>>> I'm asking is -- "dear kernel, please don't advance this process even >>> if you otherwise can". All I need is a frozen state so that I can >> >> not so easy on a multiprocessor. (unless you turn all but one >> processor off.) > > Hm. May be its getting late, but I can't quite see why that would > be the case (or may be I didn't quite communicate the intent > in my plea to the kernel ;-)) > > The target process is *already* waiting for the IO stuck inside the > kernel. It is not on a runqueue, not it is considered to be places > there. since procwrite doesn't acquire anything other than the debug lock, how do you know? the proc could start up again before you notice. you could start grabbing locks, stealing stuff off queues or adding if(up->procctl) statements here and there, but i have a strong feeling thair be dragons in that direction. the place to catch processes is on syscall entry and/or exit. stop messages are only checked on syscall entry. i have a feeling that you want something other than notes for your problem. cpu arranges that del works. - erik
Re: [9fans] Quick question on stopping a process that waits for IO
On Nov 4, 2008, at 8:00 PM, erik quanstrom wrote: i don't think the kernel has this level of control. let's suppose that we have a process that gets a stop message that's doing i/o. let's suppose that it's doing io to a particularly cranky device with lots of neat locks that really hates getting interrupted. i don't know, something esoteric — say, ata. are you really so sure that you can do what you want to this process when ata io is going on? Well, may be not. But I would like my right to shoot myself in the foot not to be revoked ;-) i sure would like to hear from someone who knows more about what went into the design of notes. i don't think i've thought of all the scary dark corners. Same here. Now, There are two issues on the table right now: the first one has everything to do with notes, and I really would like to better understand that area myself. Hence I'm all ears. The other issue (how to manipulate the running process) has less to do with notes per se. As I said, I do appreciate the reasoning that a process has to be in a frozen state in order for writes on /proc/n/mem to succeed. By a frozen state I still mean the kind of a state that would prevent a scheduler to place it on a runqueue. Stopped is one such state, but may be it shouldn't be the only one. Would it be completely unreasonable for devproc.c to also allow writes for the processes which are in process of being stopped? Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
On Nov 4, 2008, at 8:01 PM, erik quanstrom wrote: I'm asking is -- "dear kernel, please don't advance this process even if you otherwise can". All I need is a frozen state so that I can not so easy on a multiprocessor. (unless you turn all but one processor off.) Hm. May be its getting late, but I can't quite see why that would be the case (or may be I didn't quite communicate the intent in my plea to the kernel ;-)) The target process is *already* waiting for the IO stuck inside the kernel. It is not on a runqueue, not it is considered to be places there. All of the user-level data structures are frozen and not changing. All I'm asking is that if, and only if, the process gets attention from the scheduler (for whatever reason) it doesn't get placed on a runque. I don't quite see how MP would matter in this case. Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
> I'm asking is -- "dear kernel, please don't advance this process even > if you otherwise can". All I need is a frozen state so that I can not so easy on a multiprocessor. (unless you turn all but one processor off.) - erik
Re: [9fans] Quick question on stopping a process that waits for IO
> I'm glad you've asked ;-) In fact, there's a bigger context and it is > around managing processes run by cpu(1) from the terminal host. I was > planning on writing an email on that subject to this list over the > weekend but I need to amass some level of intelligence in that area > first. > > For now, though, the easiest way to illustrate my problem would be to > imagine a situation where you want to write some data into a Data > segment of the process that is waiting for the IO. Now, waiting for > the IO is almost as good as state as Stopped (the process is stuck in > kernel waiting to be put back on the runqueue) so one would expect the > transition between these 2 states to be almost trivial. Literally, all > I'm asking is -- "dear kernel, please don't advance this process even > if you otherwise can". All I need is a frozen state so that I can > manipulate some data. Yet, I can't figure out how to do that: i don't think the kernel has this level of control. let's suppose that we have a process that gets a stop message that's doing i/o. let's suppose that it's doing io to a particularly cranky device with lots of neat locks that really hates getting interrupted. i don't know, something esoteric — say, ata. are you really so sure that you can do what you want to this process when ata io is going on? i don't think this is safe unless all copies back to user space are done by some kernel function that does all the proper checking. and you have some way of unlocking safely from any part of the i/o code. i sure would like to hear from someone who knows more about what went into the design of notes. i don't think i've thought of all the scary dark corners. - erik
Re: [9fans] Quick question on stopping a process that waits for IO
On Mon, 2008-11-03 at 17:01 -0800, ron minnich wrote: > On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote: > > when somebody tries to stop a process that is waiting for the IO the process > > doesn't get transferred to a Stopped state immediately but only when > > the scheduler sees it for the first time. This leads to a process writing to > > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient. > > > > Is there any way I can poke the target process so that it gets attention > > from the scheduler an can be put in a Stopped state? > > > > Hi Roman, what's the bigger picture here? I am trying to understand > ... sorry if I'm missing the obvious. I'm glad you've asked ;-) In fact, there's a bigger context and it is around managing processes run by cpu(1) from the terminal host. I was planning on writing an email on that subject to this list over the weekend but I need to amass some level of intelligence in that area first. For now, though, the easiest way to illustrate my problem would be to imagine a situation where you want to write some data into a Data segment of the process that is waiting for the IO. Now, waiting for the IO is almost as good as state as Stopped (the process is stuck in kernel waiting to be put back on the runqueue) so one would expect the transition between these 2 states to be almost trivial. Literally, all I'm asking is -- "dear kernel, please don't advance this process even if you otherwise can". All I need is a frozen state so that I can manipulate some data. Yet, I can't figure out how to do that: term% ps | grep 310 glenda 3100:00 0:00 24K Pread cat term% cat /proc/310/seg* Stack 0f00 10000 Text R 1000 60000 Data 6000 70000 Bss 7000 70000 term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100 write: bad process or channel control request 1+0 records in 0+0 records out Of course, the process is Spleeping, so we need to stop it: term% echo stop > /proc/310/ctl This hangs, so I'll provide some input to 'cat' so that it becomes Stopped and I can do: term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100 1+0 records in 1+0 records out But! I'm really stuck if I can't give that input to something that is Pread. How are debugging/analysis tools supposed to deal with a situation like that? I would imagine this to be a pretty common situation for them, right? Thanks, Roman.
Re: [9fans] Quick question on stopping a process that waits for IO
>> Is there any way I can poke the target process so that it gets attention >> from the scheduler an can be put in a Stopped state? > > I know, I know we all don't like those guys who talk to themselves > on mailing lists replying to their own emails, but since there were > no takers here's what I cooked up so far: >echo stop > /proc/n/ctl & >echo ping > /proc/n/note >cat /proc/n/note > /dev/null > > That of course doesn't quite work either since once I resume 'n' > by issuing > echo start > /proc/n/ctl > the process fails with: > error reading : interrupted this is an uncaught error in port/proc.c:/^sleep. many interruptable calls to sleep are wrapped by while(waserror()) ; sleep(...); poperror(); but this is a catch-22. if the i/o resumed and didn't send a note, it would not be possible to interrupt processes with del with rio's current convention. on the other hand, sending a note is the only way to get to procctl unless one is doing syscall tracing. oddly procctl says /* * called splhi() by notify(). See comment in notify for the * reasoning. */ but there doesn't seem to be a comment in any version of notify i can find. > > So the question remains -- what is the proper way of putting a process > that waits for an IO into a Stopped state? i don't think it's possible without changing the kernel. but it's a good question, why does it work this way? obviously one doesn't want to stop a process and kernel work in the middle of i/o. there may be locks involved, etc. but one could easily check for Proc_stopme on syscall exit. does anyone remember the thinking that went into this? - erik
Re: [9fans] Quick question on stopping a process that waits for IO
On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik <[EMAIL PROTECTED]> wrote: > Guys, > > when somebody tries to stop a process that is waiting for the IO the process > doesn't get transferred to a Stopped state immediately but only when > the scheduler sees it for the first time. This leads to a process writing to > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient. > > Is there any way I can poke the target process so that it gets attention > from the scheduler an can be put in a Stopped state? > Hi Roman, what's the bigger picture here? I am trying to understand ... sorry if I'm missing the obvious. ron
Re: [9fans] Quick question on stopping a process that waits for IO
On Sun, 2008-11-02 at 21:55 -0800, Roman Shaposhnik wrote: > Is there any way I can poke the target process so that it gets attention > from the scheduler an can be put in a Stopped state? I know, I know we all don't like those guys who talk to themselves on mailing lists replying to their own emails, but since there were no takers here's what I cooked up so far: echo stop > /proc/n/ctl & echo ping > /proc/n/note cat /proc/n/note > /dev/null That of course doesn't quite work either since once I resume 'n' by issuing echo start > /proc/n/ctl the process fails with: error reading : interrupted So the question remains -- what is the proper way of putting a process that waits for an IO into a Stopped state? Sorry that I keep bothering the list, but I can't really think of anything here :-( Thanks, Roman.
