Re: [Chicken-users] EINTR with self-pipe signal trampoline
Thanks Mario. I just checked. The Scheduler code I'm using does not have this bug. On Sep 29 2011, Mario Domenech Goulart wrote: Hi Jörg, On 29 Sep 2011 13:11:49 +0200 Jörg F. Wittenberger wrote: Recently (maybe 4.7.3 or .4) I'm seeing missbehavior from formerly well working code. I can't say that's chickens fault but neither I can say it's not. 4.7.3 has a bug that may affect your applications: https://bugs.call-cc.org/ticket/668 ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Sep 29 2011, Jörg F. Wittenberger wrote: On Sep 29 2011, Alan Post wrote: Let me try a demonstration showing just the main thread: (define (restart-read fd buf isize) ; call read(2), on the self-pipe, which blocks (let ((r (file-read fd buf isize))) (if (= -1 r) ; ah, a signal was delivered. In Chicken, the signal ; delivery to the prcoess causes this error, but the ; Scheme code signal handler might not have been ; delivered. In this case, it means that I haven't ; written a byte to the self-pipe yet. ; (if (= errno EINTR) ; restart the read. Normally, we would have written ; to our self-pipe already (as the signal is delivered ; the moment we return from our syscall.), but in ; Chicken, I'm not sure our deferred Scheme code has ; been called. When does that happen? If it hasn't ; happened yet, I'll block in this routine, and the ; deferred signal handler will never run. ; ; I need a way to guarantee that the deferred signal ; handler has already been run, before I call any other ; syscall. ; (restart-read fd buf isize)) ; something else went wrong, die. (io-error "read" Does that story make more sense? Chicken may already behave this way, I'm not sure how long it waits/when it delivers a deferred signal. The critically important thing is that it do so before making another syscall. Yes. That looks quite like the code in the process-io-ports as I posted before. (let-location ((again bool #f)) (lambda () (when (fx>= bufindex buflen) (let loop () (and (not iclosed) (let ([n ((foreign-lambda* int ((int fd) (scheme-pointer buf) (int s) ((c-pointer bool) again)) "int r = read(fd, buf, s); *again=(r==-1)&&(errno == EAGAIN); return(r);") fdr buf buffer-size (location again))]) (cond (again (thread-wait-for-i/o! fdr #:input) (loop)) Try fixing the problem by replacing *again=(r==-1)&&(errno == EAGAIN); with *again=(r==-1)&&((errno == EAGAIN) || (errno == EINTR)); And do so again in the output case. So far I'm in a bogus situation: a) I applied that sigaction-patch. Since I've not seen the problem, which has eaten my nerves for about a week. b) still I've seen those kinda rare events, where I had some indication of i/o-threads dying in unexpected way with no good clean up. But those indications where wild guess work as you do, when you just have no clue how to force the case to happen. c) Those lost-in-io cases did not show up... within an all too short time to take the observation as serious. But still long enough to inform you of my personal feeling that there might be a connection. Alan, I'd give Felix' code a shot with the EINTR case fixed. Looks kinda good from my bogus poit of view. ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Sep 29 2011, Alan Post wrote: Let me try a demonstration showing just the main thread: (define (restart-read fd buf isize) ; call read(2), on the self-pipe, which blocks (let ((r (file-read fd buf isize))) (if (= -1 r) ; ah, a signal was delivered. In Chicken, the signal ; delivery to the prcoess causes this error, but the ; Scheme code signal handler might not have been ; delivered. In this case, it means that I haven't ; written a byte to the self-pipe yet. ; (if (= errno EINTR) ; restart the read. Normally, we would have written ; to our self-pipe already (as the signal is delivered ; the moment we return from our syscall.), but in ; Chicken, I'm not sure our deferred Scheme code has ; been called. When does that happen? If it hasn't ; happened yet, I'll block in this routine, and the ; deferred signal handler will never run. ; ; I need a way to guarantee that the deferred signal ; handler has already been run, before I call any other ; syscall. ; (restart-read fd buf isize)) ; something else went wrong, die. (io-error "read" Does that story make more sense? Chicken may already behave this way, I'm not sure how long it waits/when it delivers a deferred signal. The critically important thing is that it do so before making another syscall. Yes. That looks quite like the code in the process-io-ports as I posted before. (let-location ((again bool #f)) (lambda () (when (fx>= bufindex buflen) (let loop () (and (not iclosed) (let ([n ((foreign-lambda* int ((int fd) (scheme-pointer buf) (int s) ((c-pointer bool) again)) "int r = read(fd, buf, s); *again=(r==-1)&&(errno == EAGAIN); return(r);") fdr buf buffer-size (location again))]) (cond (again (thread-wait-for-i/o! fdr #:input) (loop)) Here the thread is scheduled to wait again if EAGAIN has been in errno. But you are right: the EINTR call is not handled. That's a bug for sure. I need to fix it. /Jörg ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Sep 29 2011, Alan Post wrote: On Thu, Sep 29, 2011 at 03:26:22PM +0200, Jörg F. Wittenberger wrote: Signal handlers as they are in chicken might be problematic. I can't find the message, but I recall a reply these days, which informed me that it's true that chicken interrupt handlers are in fact in a restricted dialect of Scheme: they can not allocate memory. I'm not sure how one would write a signal handler in chicken that doesn't allocate memory. It seems even relatively primitive signal handlers, like my example case, do so: Neither I am. That's why I have the problem with the signal handlers. I tried to do the textbook solution: just unlock a mutex in the signal handler. This would still not work for sure (though it does work for quite some time in practice, but that doesn't help in reality). (file-write output-pipe (string (integer->char signum I count that as two boxing operations, one that might not be doable So do I. In Chicken, all memory is initially allocated by moving the stack pointer to make room for the object. Upon minor gc those objects are moved from the initial position (called nursery in chicken slang) to the heap. That's why signal handlers have the problem, as I tried to explain in the last message: they are run just when space is tight. Moving them to the end of the gc (C_reclaim) *should* *probably* not hurt. Except that we *will* see a larger latency. However my attempts to do so are not yet good enough. Given that a thread or a signal handler get their own stacks, and that Chicken uses the stack as a first-generation semispace/nursery, I'm not sure why conceptually one couldn't just execute the runtime inside a signal handler. I'm afraid I'm not prepared to answer here. Attempt work around: defer the signal even more: until next schedule time. I'm not yet satisfied with that solution. (But at least it allows me to run arbitrary code in the signal handler.) (I intended so far to try whether it would be better to run it close to the finalizers; that is at the end of garbage collection. But that again is just an experiment TBD.) Interesting approach. In conversation with you, I have determined that what has to happen to preserve the execution model is that signals are delivered before a syscall is performed. It appears to me right now that any time at all between a return from a syscall and the next syscall are fine for signal delivery--that leaves a lot of room. That might be the weakness of my attempt work around! Thanks for the hint. I'll try that next. (As soon as I'm done testing your signaction patch, which just finished to compile.) ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Thu, Sep 29, 2011 at 03:26:22PM +0200, Jörg F. Wittenberger wrote: > Signal handlers as they are in chicken might be problematic. > I can't find the message, but I recall a reply these days, which > informed me that it's true that chicken interrupt handlers are > in fact in a restricted dialect of Scheme: they can not allocate memory. > I'm not sure how one would write a signal handler in chicken that doesn't allocate memory. It seems even relatively primitive signal handlers, like my example case, do so: (file-write output-pipe (string (integer->char signum I count that as two boxing operations, one that might not be doable on the stack (I don't know yet how Chicken Scheme allocates strings and therefor whether it does so on the stack. Then of course whatever memory is required to traverse the sexpr--I don't know how Chicken handles that yet either. The objects allocated could be done in advance, as you know what signal numbers you'll be receiving, but I've just assumed the deferral mechanism for signals is done in part to allow memory to be allocated inside them, because they're called in the main thread (or some rough equivalent). *C* signal handlers can't allocate memory, for sure--I'd consider it poor form to do something in a Chicken signal handler that isn't adviseable from a C signal handler, but the runtime environment makes that boundary fuzzy, for me. > I started to read into it. I too, still don't understand all of the > C_reclaim procedure. But it's obvious that the interrupt handler > is invoked at the begin of the garbage collection. Wild guess: > there is no memory available for allocation at that time. > Given that a thread or a signal handler get their own stacks, and that Chicken uses the stack as a first-generation semispace/nursery, I'm not sure why conceptually one couldn't just execute the runtime inside a signal handler. There are things I'd forbid: no signals delivered during GC, no GC in the signal handler, &c. It may well be nothing but hair all the way down, but architecturally I haven't imagined anything to prevent it. > Attempt work around: defer the signal even more: until next schedule > time. I'm not yet satisfied with that solution. (But at least it > allows me to run arbitrary code in the signal handler.) > (I intended so far to try whether it would be better to run it close > to the finalizers; that is at the end of garbage collection. > But that again is just an experiment TBD.) > Interesting approach. In conversation with you, I have determined that what has to happen to preserve the execution model is that signals are delivered before a syscall is performed. It appears to me right now that any time at all between a return from a syscall and the next syscall are fine for signal delivery--that leaves a lot of room. .i ko .e mi ze'aca kelgu'a (let's keep hacking), -Alan -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Thu, Sep 29, 2011 at 03:26:22PM +0200, Jörg F. Wittenberger wrote: > Attempt work around: defer the signal even more: until next schedule > time. I'm not yet satisfied with that solution. (But at least it > allows me to run arbitrary code in the signal handler.) > (I intended so far to try whether it would be better to run it close > to the finalizers; that is at the end of garbage collection. > But that again is just an experiment TBD.) > > Given your situation I hope we can devise a better way. > > I'd need to understand your situation better. > > >I need a way to deliver deferred signals after a syscall returns > >EINTR, before restarting that syscall. > > I see. I understand you need "minimum latency" - right? > > May I ask for more details. I need to understand where this > latency requirement is important. > Let me try a demonstration showing just the main thread: (define (restart-read fd buf isize) ; call read(2), on the self-pipe, which blocks (let ((r (file-read fd buf isize))) (if (= -1 r) ; ah, a signal was delivered. In Chicken, the signal ; delivery to the prcoess causes this error, but the ; Scheme code signal handler might not have been ; delivered. In this case, it means that I haven't ; written a byte to the self-pipe yet. ; (if (= errno EINTR) ; restart the read. Normally, we would have written ; to our self-pipe already (as the signal is delivered ; the moment we return from our syscall.), but in ; Chicken, I'm not sure our deferred Scheme code has ; been called. When does that happen? If it hasn't ; happened yet, I'll block in this routine, and the ; deferred signal handler will never run. ; ; I need a way to guarantee that the deferred signal ; handler has already been run, before I call any other ; syscall. ; (restart-read fd buf isize)) ; something else went wrong, die. (io-error "read" Does that story make more sense? Chicken may already behave this way, I'm not sure how long it waits/when it delivers a deferred signal. The critically important thing is that it do so before making another syscall. -Alan -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Sep 29 2011, Alan Post wrote: On Thu, Sep 29, 2011 at 01:11:49PM +0200, Jörg F. Wittenberger wrote: On Sep 29 2011, Alan Post wrote: >Below is a test case for a problem I'm seeing in some multi-process >code I'm writing. I'm getting the error: > > Error: (file-read) cannot read from file - Interrupted system call There are two ways to fix that: either make the posix unit thread safe (recall my recent message how to avoid process-wait having a bad effect). The other one is working around the problem. That's what I'm doing based on some code Felix supplied ages ago. It wraps the file descriptors into custom ports those are properly restarted on EINTR. However I'd be rather interested to learn what exactly the problem is you observe. Recently (maybe 4.7.3 or .4) I'm seeing missbehavior from formerly well working code. I can't say that's chickens fault but neither I can say it's not. If I understand the part of the code below that wraps read/write, it can't be used as-is for my problem. Because chicken defers signals, a signal is delivered, deferred, then read/write return with EINTR. Ah, I see! Good catch. If I immediately restart these syscalls (all in the same C call), the deferred signal has not be delivered, and the code deadlocks, as read/write pauses--blocking the signal handler from ever being run. I'm not yet sure, but this could shed some light at the problems I have. However I normally see signal handling working all day long in my code. Therefore I don't fully understand your issue yet. Signal handlers as they are in chicken might be problematic. I can't find the message, but I recall a reply these days, which informed me that it's true that chicken interrupt handlers are in fact in a restricted dialect of Scheme: they can not allocate memory. I started to read into it. I too, still don't understand all of the C_reclaim procedure. But it's obvious that the interrupt handler is invoked at the begin of the garbage collection. Wild guess: there is no memory available for allocation at that time. Attempt work around: defer the signal even more: until next schedule time. I'm not yet satisfied with that solution. (But at least it allows me to run arbitrary code in the signal handler.) (I intended so far to try whether it would be better to run it close to the finalizers; that is at the end of garbage collection. But that again is just an experiment TBD.) Given your situation I hope we can devise a better way. I'd need to understand your situation better. I need a way to deliver deferred signals after a syscall returns EINTR, before restarting that syscall. I see. I understand you need "minimum latency" - right? May I ask for more details. I need to understand where this latency requirement is important. You may well not notice this in non-blocking code, as no data would be ready on the file descriptor and the code would continue running--eventually to deliver the deferred signal and unwedge everything Sorry. I do not have so many chicken programs. As soon as something blocks too often, I'll noticed a heavy slow down so far. Under normal circumstances my prog uses 100-200 file descriptors connected to pipes to subprocesses (which do network i/o+ssl) and always on self-pipe. Plus a few open sockets. /Jörg ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
Hi Jörg, On 29 Sep 2011 13:11:49 +0200 Jörg F. Wittenberger wrote: > Recently (maybe 4.7.3 or .4) I'm seeing missbehavior from formerly > well working code. I can't say that's chickens fault but neither I > can say it's not. 4.7.3 has a bug that may affect your applications: https://bugs.call-cc.org/ticket/668 Best wishes. Mario -- http://parenteses.org/mario ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Thu, Sep 29, 2011 at 01:11:49PM +0200, Jörg F. Wittenberger wrote: > On Sep 29 2011, Alan Post wrote: > > >Below is a test case for a problem I'm seeing in some multi-process > >code I'm writing. I'm getting the error: > > > > Error: (file-read) cannot read from file - Interrupted system call > > There are two ways to fix that: either make the posix unit thread safe > (recall my recent message how to avoid process-wait having a bad effect). > > The other one is working around the problem. That's what I'm doing based > on some code Felix supplied ages ago. It wraps the file descriptors > into custom ports those are properly restarted on EINTR. > > However I'd be rather interested to learn what exactly the problem is > you observe. Recently (maybe 4.7.3 or .4) I'm seeing missbehavior > from formerly well working code. I can't say that's chickens fault > but neither I can say it's not. > If I understand the part of the code below that wraps read/write, it can't be used as-is for my problem. Because chicken defers signals, a signal is delivered, deferred, then read/write return with EINTR. If I immediately restart these syscalls (all in the same C call), the deferred signal has not be delivered, and the code deadlocks, as read/write pauses--blocking the signal handler from ever being run. I need a way to deliver deferred signals after a syscall returns EINTR, before restarting that syscall. You may well not notice this in non-blocking code, as no data would be ready on the file descriptor and the code would continue running--eventually to deliver the deferred signal and unwedge everything. -Alan -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Wed, Sep 28, 2011 at 05:29:46PM -0601, Alan Post wrote: > Below is a test case for a problem I'm seeing in some multi-process > code I'm writing. I'm getting the error: > > Error: (file-read) cannot read from file - Interrupted system call > > because a signal handler is going off while my main thread is in an > iowait state. In C, I have always handled this by manually > restarting a system call when EINTR is called. In my attached test > case, I'm trying to use SA_RESTART in sigaction(2) for the same > affect, and that isn't working for me. Does that actually work? > > I'm trying to determine which direction to head in to fix this > problem, and I was surprised that SA_RESTART didn't work. Will one > of you with access to a linux machine try running this code? Both > in the version as seen here and also run uncommenting the two > "foreign-code" lines. > > This code might have bugs that prevent it from running as > intended--I'm not able to test past my EINTR problem. I'm now > planning on patching file-read and file-write to restart on EINTR, > unless someone has a better idea! Can I catch this exception at > runtime? > > -Alan > I investigated this further last night and can articulate better the problem. 1) The main thread sets a signal handler. 2) In then calls read(2) and blocks. 3) A signal arrives, which is handled by an internal signal handler. 4) read(2) returns -1 with errno set to EINTR. What I need to have happen here, is for the signal handling code to then be run, which will write(2) to the pipe. I then need the signal handler to be run (which happens on the main thread) At which point I *then* need to restart the read(2) call, which now will have pending data. I can't introduce a version of read(2) that restarts on EINTR: doing that causes the signal handler to never be run, so we pause forever in a deadlock. I don't understand enough about the scheduler to know when interrupts get handled. Can it happen while executing the scheme code in file-read? If signals get delivered at a reliable point, read can likely be restarted knowing the signal has been handled. If not, I'm left with needing to catch the error, somehow guarantee that the signal handler has been run, then call my code again (to re-enter read) When does the scheduler handle interrupts? -Alan -- .i ma'a lo bradi cu penmi gi'e du ___ Chicken-users mailing list Chicken-users@nongnu.org https://lists.nongnu.org/mailman/listinfo/chicken-users
Re: [Chicken-users] EINTR with self-pipe signal trampoline
On Sep 29 2011, Alan Post wrote: Below is a test case for a problem I'm seeing in some multi-process code I'm writing. I'm getting the error: Error: (file-read) cannot read from file - Interrupted system call There are two ways to fix that: either make the posix unit thread safe (recall my recent message how to avoid process-wait having a bad effect). The other one is working around the problem. That's what I'm doing based on some code Felix supplied ages ago. It wraps the file descriptors into custom ports those are properly restarted on EINTR. However I'd be rather interested to learn what exactly the problem is you observe. Recently (maybe 4.7.3 or .4) I'm seeing missbehavior from formerly well working code. I can't say that's chickens fault but neither I can say it's not. Here is the code. Please excuse that uses process-wait-for-pid and process-test-pid to avoid blocking for the childs termination. You might replace them with process-wait whereby process-test-pid should never block. (define process-io-ports (let ([make-input-port make-input-port] [make-output-port make-output-port] [make-string make-string] [substring substring] [file-close-fd! (foreign-lambda* int ((int fd)) "return(close(fd));")] ) (lambda (pid fdr fdw) (##sys#file-nonblocking! fdw) (##sys#file-nonblocking! fdr) ; should not be required (let* ([buf (make-string buffer-size)] ;; [data (vector #f #f #f)] [buflen 0] [bufindex 0] [iclosed #f] [oclosed #f] [in (make-input-port (let-location ((again bool #f)) (lambda () (when (fx>= bufindex buflen) (let loop () (and (not iclosed) (let ([n ((foreign-lambda* int ((int fd) (scheme-pointer buf) (int s) ((c-pointer bool) again)) "int r = read(fd, buf, s); *again=(r==-1)&&(errno == EAGAIN); return(r);") fdr buf buffer-size (location again))]) (cond (again (thread-wait-for-i/o! fdr #:input) (loop)) ((eq? -1 n) ;; (##sys#update-errno) ;; (##sys#signal-hook #:process-error 'process-io-read "can not read from fd" fdr strerror) (set! iclosed #t) #;(when (eq? -1 (file-close-fd! fdr)) (##sys#update-errno) (##sys#signal-hook #:process-error 'process-io-read "can not close fd input port" fdr) ) (file-close-fd! fdr)) (else ;; (print "[rd: " n "]") (set! buflen n) (set! bufindex 0))) ))) ) (if (or iclosed (fx>= bufindex buflen)) (end-of-file) (let ([c (##core#inline "C_subchar" buf bufindex)]) (set! bufindex (fx+ bufindex 1)) c) ) )) (lambda () (when iclosed (##sys#signal-hook #:process-error "input port is closed" fdr)) #t ) (lambda () (unless iclosed (set! iclosed #t) (when (eq? -1 (file-close-fd! fdr)) (##sys#update-errno) (##sys#signal-hook #:process-error 'process-io-close "can not close fd input port" fdr) ) (if oclosed (receive (p f s) (process-wait-for-pid pid) s) (receive (p f s) (process-test-pid pid) (when (eqv? p pid) (set! oclosed #t) (when (eq? -1 (file-close-fd! fdw)) (##sys#update-errno) (##sys#signal-hook #:process-error 'process-io-close "can not close fd output port" fdw) ) s))) ) ) ) ] [out (let-location ((again bool #f)) (make-output-port (lambda (s) (define start-time (current-milliseconds)) (let loop ([len (##sys#size s)] [off 0]) (if oclosed (##sys#signal-hook
