Re: NSTask oddity with getting stdout
On Jul 26, 2011, at 11:22 PM, Scott Ribe wrote: > On Jul 26, 2011, at 5:52 PM, Shane Stanley wrote: > >> In Snow Leopard that worked fine; a notification would be sent when new data >> was written to the file. In Lion, as soon as it's called it goes into a >> loop; each time readInBackgroundAndNotify is sent, a new notification comes >> straight back. > > An actual file? I'm reading from a pipe, so that could be one difference. Yes. For a file (vnode), there's no such thing as non-blocking or asynchronous I/O through the typical BSD/POSIX APIs (ignoring aio). All attempts to read or write will complete synchronously, blocking even if the file descriptor has been configured as non-blocking and the file system is mounted over a slow or broken network connection. Attempts to test a vnode file descriptor's readability or writability using select() or poll() will always report that the file is readable or writable. kqueue() is a bit weird for vnodes. EVFILT_READ fires whenever the file pointer is not at the end of the file. That means you'll be notified repeatedly until you read to the end of the file, but won't be notified when you're actually at the end of the file. That is, you won't be told to read once more to receive the EOF indication (e.g. zero-length read). After that, you won't be notified again unless and until the file is extended by something writing more data to it, past the current end, or if you seek to some other position. Although this seems to be what Shane wants, for most programmers this is a surprising result and differs quite a lot from other types of file descriptors. It also doesn't correspond to NSFileHandle's documented behavior, since that doesn't differ by descriptor type. Kqueue's EVFILT_WRITE isn't supported at all for vnodes, as it doesn't make sense, since it would always fire immediately. The caller should simply not use kqueue() and should just go ahead and write unconditionally, instead. None of this applies for I/O through a pipe or socket. Regards, Ken ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On Jul 26, 2011, at 5:52 PM, Shane Stanley wrote: >> Maybe you should try to re-open your bug ;-) > > I did, but no reply. Too bad, since it seems to me the person who responded didn't know what they were talking about. > In Snow Leopard that worked fine; a notification would be sent when new data > was written to the file. In Lion, as soon as it's called it goes into a > loop; each time readInBackgroundAndNotify is sent, a new notification comes > straight back. An actual file? I'm reading from a pipe, so that could be one difference. I would expect that to go into a loop once a data of length 0 comes in, but not before. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On 27/7/11 3:18 AM, "Scott Ribe" wrote: > Maybe you should try to re-open your bug ;-) I did, but no reply. Here's some code that's reading a file that's being written periodically: -(void)dataFromFile:(NSNotification *)notif { NSData *data = [[notif userInfo] objectForKey:NSFileHandleNotificationDataItem]; if ([data length]) { // do stuff } [[notif object] readInBackgroundAndNotify]; // call again } In Snow Leopard that worked fine; a notification would be sent when new data was written to the file. In Lion, as soon as it's called it goes into a loop; each time readInBackgroundAndNotify is sent, a new notification comes straight back. -- Shane Stanley 'AppleScriptObjC Explored' ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On Jul 26, 2011, at 12:18 AM, Shane Stanley wrote: >> On Jul 25, 2011, at 3:40 AM, Shane Stanley wrote: >> I wonder if they've changed how the readInBackgroundAndNotify works. >>> >>> Yes, they have. I logged a bug on it, and was told that the way it worked in >>> 10.6 was wrong -- whereas it used not to send a notification until there was >>> something to read, it now returns one immediately regardless. Whether that's >>> the problem here, I don't know. >> >> That's absolutely demented. If it sends a notification immediately, with or >> without data, then you're never really using it to read in the background, >> you're almost just spinning in a busy loop, getting a probably empty NSData, >> and calling readInBackgroundAndNotify again, as fast as the runloop can >> deliver the notifications. > > Exactly. I built a sample app that did exactly that, and logged radar > #9120065. I was told that was correct behavior, and that "Snow Leopard's > NSFileHandle never correctly notified with a zero length data at EOF". The > suggested fix was "do a blocking -readDataOfLength: call on a background > thread". - Snow Leopard most certainly did correctly notify with a zero length data at EOF for me and did so sufficiently consistently that I am not aware of it ever failing to do so, in thousands of runs. Therefore "never" is clearly incorrect. - I don't see how what you said "...it now returns one immediately regardless" follows from Apple's response that you quoted; maybe I'm missing some context. - What good will a blocking readDataOfLength: call on a background thread do for the case where you don't know what's coming, need to process it as it comes, then need to know when it's closed? Are they seriously suggesting that we need to read a single character at a time using blocking reads on a background thread? That's such a lame suggestion that it would make me ask: why bother with NSTask and NSFileHandle and notifications at all? In that case POSIX APIs would be no harder to use, and would actually work. - I set up some tests in Lion and ran them last night, and never had an instance of getting a notification with zero-length data before the pipe was closed. Only a few dozen runs, but still, 0-length data always indicated pipe close. (I modified the command-line process with a couple of obnoxious pauses, before sending anything and before sending the final lines, so that if there's some timeout it would be likely to be hit.) Maybe you should try to re-open your bug ;-) -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On 26/7/11 1:14 AM, "Scott Ribe" wrote: > On Jul 25, 2011, at 3:40 AM, Shane Stanley wrote: > >>> I wonder if they've changed how the readInBackgroundAndNotify works. >> >> Yes, they have. I logged a bug on it, and was told that the way it worked in >> 10.6 was wrong -- whereas it used not to send a notification until there was >> something to read, it now returns one immediately regardless. Whether that's >> the problem here, I don't know. > > That's absolutely demented. If it sends a notification immediately, with or > without data, then you're never really using it to read in the background, > you're almost just spinning in a busy loop, getting a probably empty NSData, > and calling readInBackgroundAndNotify again, as fast as the runloop can > deliver the notifications. Exactly. I built a sample app that did exactly that, and logged radar #9120065. I was told that was correct behavior, and that "Snow Leopard's NSFileHandle never correctly notified with a zero length data at EOF". The suggested fix was "do a blocking -readDataOfLength: call on a background thread". -- Shane Stanley 'AppleScriptObjC Explored' ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On Jul 25, 2011, at 3:40 AM, Shane Stanley wrote: >> I wonder if they've changed how the readInBackgroundAndNotify works. > > Yes, they have. I logged a bug on it, and was told that the way it worked in > 10.6 was wrong -- whereas it used not to send a notification until there was > something to read, it now returns one immediately regardless. Whether that's > the problem here, I don't know. That's absolutely demented. If it sends a notification immediately, with or without data, then you're never really using it to read in the background, you're almost just spinning in a busy loop, getting a probably empty NSData, and calling readInBackgroundAndNotify again, as fast as the runloop can deliver the notifications. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On Jul 24, 2011, at 11:01 PM, Stephen J. Butler wrote: > That is, abandon processPipeClose and let each pipe handle its own > close in the notification when you're sure all of its read > notifications have actually been delivered. As you've presented it, > processPipeClose isn't needed. Your theory makes sense to me, and I think for now I'll assume it's the explanation for the behavior. In the real code there's things I need to do after receipt of complete output from both pipes; I just stripped that out of processPipeClose to make this easier to read. However, I can still incorporate your suggestion, and just do the "after everything" processing when the second of curStdOut & curStdErr go to nil. If I weren't feeling lazy this morning, I would dig back through version control. I think the code some time ago was more similar to your suggestion, but loaded up with some unnecessary complications, and this race condition is a result of a simplification attempt which I did not get quite right. Thanks a lot. This was, IMO, not a trivial question ;-) -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On 25/7/11 3:01 PM, "Stephen J. Butler" wrote: > I wonder if they've changed how the readInBackgroundAndNotify works. Yes, they have. I logged a bug on it, and was told that the way it worked in 10.6 was wrong -- whereas it used not to send a notification until there was something to read, it now returns one immediately regardless. Whether that's the problem here, I don't know. -- Shane Stanley 'AppleScriptObjC Explored' ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSTask oddity with getting stdout
On Sun, Jul 24, 2011 at 10:17 PM, Scott Ribe wrote: > I'm using NSTask to run a background process, monitor its output, and > ultimately present a message to the user. All has been fine with this 10.2 > through 10.6, as far as I know. Now under 10.7 I saw a case where my code did > not receive quite all the stdout output from the background process. I saw > this once; I haven't tried running it a huge number of times, but it is > definitely not the common case. (And I've never seen it in dev mode running > under the debugger, only on a machine without dev tools. Also BTW, the output > of the bg process is plain ASCII, no worries about multi-byte sequences.) > Does anybody see anything wrong with this code (severely stripped for > legibility): Interesting. THIS IS JUST ME SPECULATING OUT LOUD. I wonder if they've changed how the readInBackgroundAndNotify works. In particular, my guess here would be that: 1) Two "threads" are created to handle the stdout and stderr (may be dispatch queues, but let's say threads for this thought experiment). 2) Your task generates output in stdout, and most of it gets consumed. 3) Your task generates its last data and closes its stderr then stdout. 4) The stdout thread reads the last data and readies a notification, but doesn't get a chance to post it before being swapped. 5) The stderr thread sees a closed socket and posts a notification before being swapped. 6) The stdout thread wakes up, and posts its notification. Now you are in a situation where there are two notifications being delivered, and their order is wrong for your code. Namely, when the stderr notification arrives first you will do this: a) see that stderr is closed. b) try to read all the remaining data from stdout. There is none, because it has been queued already. c) remove yourself as a listener for stdout. d) try to read all the remaining data from stderr. e) remove yourself as a listener for stderr. The problem is step (c)! There might be data in the notification queue that you will never receive. Here's how I'd write your methods: > - (void) doSomeTask: (NSTask *) task Same. > - (void) processStdOut: (NSNotification *) ntc > { > NSFileHandle *f = [ntc object]; > NSData *d = [[ntc userInfo] objectForKey: > NSFileHandleNotificationDataItem]; > > if( [d length] == 0 ) { [[NSNotificationCenter defaultCenter] removeObserver: self name: NSFileHandleReadCompletionNotification object: curStdOut]; curStdOut = nil; } > else > { > [f readInBackgroundAndNotify]; > [curStdOutStr appendString: > [[[NSString alloc] initWithData: d encoding: > NSASCIIStringEncoding] autorelease]]; > } > } > > > - (void) processStdErr: (NSNotification *) ntc > { > NSFileHandle *f = [ntc object]; > NSData *d = [[ntc userInfo] objectForKey: > NSFileHandleNotificationDataItem]; > > if( [d length] == 0 ) { [[NSNotificationCenter defaultCenter] removeObserver: self name: NSFileHandleReadCompletionNotification object: curStdErr]; curStdErr = nil; } > else > { > [f readInBackgroundAndNotify]; > [curStdErrStr appendString: > [[[NSString alloc] initWithData: d encoding: > NSASCIIStringEncoding] autorelease]]; > } > } That is, abandon processPipeClose and let each pipe handle its own close in the notification when you're sure all of its read notifications have actually been delivered. As you've presented it, processPipeClose isn't needed. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
NSTask oddity with getting stdout
I'm using NSTask to run a background process, monitor its output, and ultimately present a message to the user. All has been fine with this 10.2 through 10.6, as far as I know. Now under 10.7 I saw a case where my code did not receive quite all the stdout output from the background process. I saw this once; I haven't tried running it a huge number of times, but it is definitely not the common case. (And I've never seen it in dev mode running under the debugger, only on a machine without dev tools. Also BTW, the output of the bg process is plain ASCII, no worries about multi-byte sequences.) Does anybody see anything wrong with this code (severely stripped for legibility): - (void) doSomeTask: (NSTask *) task { NSTask * task = [[NSTask alloc] init]; [task setLaunchPath: cmdpath]; [task setArguments: [self getArgs]]; [task setStandardError: [NSPipe pipe]]; [task setStandardOutput: [NSPipe pipe]]; [task launch]; curScanTask = [task retain]; curStdOutStr = [[NSMutableString alloc] init]; curStdErrStr = [[NSMutableString alloc] init]; curStdOut = [[task standardOutput] fileHandleForReading]; curStdErr = [[task standardError] fileHandleForReading]; [[NSNotificationCenter defaultCenter] addObserver: self selector: @selector( processStdOut: ) name: NSFileHandleReadCompletionNotification object: curStdOut]; [[NSNotificationCenter defaultCenter] addObserver: self selector: @selector( processStdErr: ) name: NSFileHandleReadCompletionNotification object: curStdErr]; [curStdOut readInBackgroundAndNotify]; [curStdErr readInBackgroundAndNotify]; } - (void) processStdOut: (NSNotification *) ntc { NSFileHandle *f = [ntc object]; NSData *d = [[ntc userInfo] objectForKey: NSFileHandleNotificationDataItem]; if( [d length] == 0 ) [self processPipeClose]; else { [f readInBackgroundAndNotify]; [curStdOutStr appendString: [[[NSString alloc] initWithData: d encoding: NSASCIIStringEncoding] autorelease]]; } } - (void) processStdErr: (NSNotification *) ntc { NSFileHandle *f = [ntc object]; NSData *d = [[ntc userInfo] objectForKey: NSFileHandleNotificationDataItem]; if( [d length] == 0 ) [self processPipeClose]; else { [f readInBackgroundAndNotify]; [curStdErrStr appendString: [[[NSString alloc] initWithData: d encoding: NSASCIIStringEncoding] autorelease]]; } } - (void) processPipeClose { [curStdOutStr appendString: [[[NSString alloc] initWithData: [curStdOut readDataToEndOfFile] encoding: NSASCIIStringEncoding] autorelease]]; [[NSNotificationCenter defaultCenter] removeObserver: self name: NSFileHandleReadCompletionNotification object: curStdOut]; curStdOut = nil; [curStdErrStr appendString: [[[NSString alloc] initWithData: [curStdErr readDataToEndOfFile] encoding: NSASCIIStringEncoding] autorelease]]; [[NSNotificationCenter defaultCenter] removeObserver: self name: NSFileHandleReadCompletionNotification object: curStdErr]; curStdErr = nil; } -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com