Re: Notification of file system modification arrives too early?
Le 31 mai 2010 à 05:39, Dave Keck a écrit : Unfortunately you probably can’t do any better than that, since there’s no cheap way to find out if another process has the file open. proc_listpidspath() is meant for this, but it is indeed quite expensive. In my testing, it takes about a second to complete this call; furthermore, its status as a supported API is unclear. Really ? I find the first comment in libproc.h pretty clear about it. /* * This header file contains private interfaces to obtain process information. * These interfaces are subject to change in future releases. */ -- Jean-Daniel ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
Really ? I find the first comment in libproc.h pretty clear about it. If it was private in the strictest sense, the header wouldn't exist. Certainly since the header has existed since 10.5, Apple intends for someone to use it? ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 31 May 2010, at 12:20, Dave Keck wrote: Really ? I find the first comment in libproc.h pretty clear about it. If it was private in the strictest sense, the header wouldn't exist. Certainly since the header has existed since 10.5, Apple intends for someone to use it? I doubt it. Head over to darwin-dev and ask, and I think you'll be told that you shouldn't be using it. That's even the case for some POSIX-defined APIs, actually, because they rely on specifics that may change even in a minor version update. The new version of iDefrag includes code to quickly detect whether a file is currently open or not, and to do it we had to write a kext. Kind regards, Alastair. -- http://alastairs-place.net ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On May 31, 2010, at 4:20 AM, Dave Keck wrote: If it was private in the strictest sense, the header wouldn't exist. Keep in mind that the kernel and much of the Darwin-level code is open source, so it has no such thing as a private API in the strictest sense. In open-source projects there are often APIs that happen to exist in headers but are considered unstable and not suitable for general use. —Jens ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
Antonio Nunes mailto:devli...@sintraworks.com wrote (Friday, May 28, 2010 3:05 AM +0100): My app is set to observe a folder for changes to its contents (using UKKQueue. Thanks Uli! :-)). When files are dropped into the folder the app is notified and processes the files. This works well, except for one catch: sometimes we receive the notification and spawn the worker thread, and trying to load the file fails ( [[PDFDocument alloc] initWithURL:sourceURL] returns nil). This seems to happen predominantly with large files. I get the impression the notification happens as soon as the file transfer into the watched directory begins, and my worker thread is ready and starts loading the file before it has been fully transferred. Hence the failure to create a PDFDocument from the file. Is there a way to check whether the file has been fully transferred, or, even better, to get notified only when the file transfer has been completed? I deal with this in my app by creating a kqueue to watch for changes in the enclosing folder. When a change occurs, I attempt to open the file using the FS API specifying exclusive read + exclusive write access. If it fails, it's likely because some other process still has the file open. I delay a half second or so and try again. -- James Bucanek ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On May 30, 2010, at 7:57 AM, James Bucanek wrote: When a change occurs, I attempt to open the file using the FS API specifying exclusive read + exclusive write access. If it fails, it's likely because some other process still has the file open. I delay a half second or so and try again. That sounds like a good approach, but it’s not 100% reliable. If the creating process opened the file without requesting an exclusive lock, then your open call will succeed even though the file is still open. Unfortunately you probably can’t do any better than that, since there’s no cheap way to find out if another process has the file open. —Jens___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 30 May 2010, at 17:51, Jens Alfke wrote: When a change occurs, I attempt to open the file using the FS API specifying exclusive read + exclusive write access. If it fails, it's likely because some other process still has the file open. I delay a half second or so and try again. That sounds like a good approach, but it’s not 100% reliable. If the creating process opened the file without requesting an exclusive lock, then your open call will succeed even though the file is still open. Unfortunately you probably can’t do any better than that, since there’s no cheap way to find out if another process has the file open. Thanks guys, that makes it sound like my previously mentioned solution/work-around of trying to create a PDFDocument from the URL is the safest solution. The attempt will fail until the file has been fully written, and I only need to read the data, not write to it. Also, as far as I can tell, reading a PDFDocument with initWithURL: (and immediately discarding it if it succeeds) is a fairly cheap operation. António - Forgiveness is not an occasional act; it is a permanent attitude. --Martin Luther King, Jr - ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
Unfortunately you probably can’t do any better than that, since there’s no
cheap way to find out if another process has the file open.
proc_listpidspath() is meant for this, but it is indeed quite expensive. In
my testing, it takes about a second to complete this call; furthermore, its
status as a supported API is unclear. Nonetheless, here's some
somewhat-tested code of how to use it:
==
#import libproc.h
- (NSSet *)pidsAccessingPath: (NSString *)path
{
const char *pathFileSystemRepresentation = nil;
int listpidspathResult = 0;
size_t pidsSize = 0;
pid_t *pids = nil;
NSUInteger pidsCount = 0,
i = 0;
NSMutableSet *result = nil;
NSParameterAssert(path [path length]);
pathFileSystemRepresentation = [path GCSafeFileSystemRepresentation];
listpidspathResult = proc_listpidspath(PROC_ALL_PIDS, 0,
pathFileSystemRepresentation, PROC_LISTPIDSPATH_EXCLUDE_EVTONLY, nil, 0);
ALAssertOrPerform(listpidspathResult = 0, goto cleanup);
pidsSize = (listpidspathResult ? listpidspathResult : 1);
pids = malloc(pidsSize);
ALAssertOrPerform(pids, goto cleanup);
listpidspathResult = proc_listpidspath(PROC_ALL_PIDS, 0,
pathFileSystemRepresentation, PROC_LISTPIDSPATH_EXCLUDE_EVTONLY, pids,
pidsSize);
ALAssertOrPerform(listpidspathResult = 0, goto cleanup);
pidsCount = (listpidspathResult / sizeof(*pids));
result = [NSMutableSet set];
for (i = 0; i pidsCount; i++)
[result addObject: [NSNumber numberWithInt: pids[i]]];
cleanup:
{
if (pids)
free(pids),
pids = nil;
}
return result;
}
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
Jens Alfke mailto:j...@mooseyard.com wrote (Sunday, May 30, 2010 9:51 AM -0700): On May 30, 2010, at 7:57 AM, James Bucanek wrote: When a change occurs, I attempt to open the file using the FS API specifying exclusive read + exclusive write access. If it fails, it's likely because some other process still has the file open. I delay a half second or so and try again. That sounds like a good approach, but it’s not 100% reliable. If the creating process opened the file without requesting an exclusive lock, then your open call will succeed even though the file is still open. It isn't 100%, but not for that reason. When you call FSOpenFork and specify fsWrDenyPerm, it will fail if another process currently has that fork open for writing. The other process does not have to request exclusive write access, just write access, which is probably the case for a file that's being written. From the API docs for fsWrDenyPerm: ... the path cannot be opened if you request deny-write permission, but some other path already has write access. It isn't 100% because not all volumes support deny-read and deny-write modes. See bHasOpenDeny. -- James Bucanek ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On May 28, 2010, at 12:05 PM, Antonio Nunes wrote: My app is set to observe a folder for changes to its contents (...) I get the impression the notification happens as soon as the file transfer into the watched directory begins, and my worker thread is ready and starts loading the file before it has been fully transferred. Hence the failure to create a PDFDocument from the file. Yes. A KQueue only tells you when a *change* happens, i.e. an individual write operation. Is there a way to check whether the file has been fully transferred, or, even better, to get notified only when the file transfer has been completed? The heuristic most people use is to wait a little time (a few seconds is usually enough) and only start processing a file if a new write notification hasn't arrived in the meantime. I usually use my UKPushbackMessenger for that purpose (see http://github.com/uliwitness/UliKit/blob/master/UKPushbackMessenger.h and http://github.com/uliwitness/UliKit/blob/master/UKPushbackMessenger.m). -- Uli Kusterer The Witnesses of TeachText are everywhere... ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 28 May 2010, at 11:30, Uli Kusterer wrote: The heuristic most people use is to wait a little time (a few seconds is usually enough) and only start processing a file if a new write notification hasn't arrived in the meantime. I usually use my UKPushbackMessenger for that purpose (see http://github.com/uliwitness/UliKit/blob/master/UKPushbackMessenger.h andhttp://github.com/uliwitness/UliKit/blob/master/UKPushbackMessenger.m). Gruezi Uli, I was hoping the system would provide something better for this. Looks like an enhancement request is in order. Since PDF documents can potentially be huge, and I need to keep performance as good as possible, I need something more robust than waiting a few seconds. I am now experimenting with trying to read in a PDFDocument from the url, and not spawn the worker thread until this succeeds. Initial results look promising. The thread that sets up the task waits until it can successfully read in a PDFDocument from the URL, then releases the document and launches the task. I find this a bit of a hack, but if it works, it works, and I think this provides tighter performance than waiting for a set number of seconds (which may in extreme cases not to be long enough anyway). Best, António --- Don't believe everything you think --- ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 28 May 2010, at 11:41, Antonio Nunes wrote: I was hoping the system would provide something better for this. Looks like an enhancement request is in order. Request filed. Bug ID# 8038793: Need notification of file system modification after completion of the operation. Cheers, António There is nothing as strong as real gentleness, and there is nothing as gentle as real strength. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On May 28, 2010, at 12:53 PM, Antonio Nunes wrote: On 28 May 2010, at 11:41, Antonio Nunes wrote: I was hoping the system would provide something better for this. Looks like an enhancement request is in order. Request filed. Bug ID# 8038793: Need notification of file system modification after completion of the operation. Are you doing this for Finder-originated copying only? In this case, you could probably look at the file's OSType/creator. I believe they get set to some special busy-values during copying, so if you get a write notification and your file has that type, you *know* it's not intended to be processed yet. -- Uli Kusterer The Witnesses of TeachText are everywhere... ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
Le 28 mai 2010 à 12:53, Antonio Nunes a écrit : On 28 May 2010, at 11:41, Antonio Nunes wrote: I was hoping the system would provide something better for this. Looks like an enhancement request is in order. Request filed. Bug ID# 8038793: Need notification of file system modification after completion of the operation. I hope you properly defined what you mean by operation, because on a file system point of view, this is already what kqueue does. A copy is not one operation but a bunch of operations (open, lots of write, close, set attrs, set xattrs, …), and kqueue already notify you each time an operation is done. -- Jean-Daniel ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 28 May 2010, at 12:15, Jean-Daniel Dupas wrote: I hope you properly defined what you mean by operation, because on a file system point of view, this is already what kqueue does. A copy is not one operation but a bunch of operations (open, lots of write, close, set attrs, set xattrs, …), and kqueue already notify you each time an operation is done. Hmm, I see, better have another good look at the notifications the I can choose to listen to. Thanks. Looks like an attribute change notification and a size increase notification might be useful candidates. I'll play with them and see what results I can get. António A merry heart does good like medicine ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On 28 May 2010, at 12:07, Uli Kusterer wrote: Are you doing this for Finder-originated copying only? In this case, you could probably look at the file's OSType/creator. I believe they get set to some special busy-values during copying, so if you get a write notification and your file has that type, you *know* it's not intended to be processed yet. I expect many of the files will arrive in the watched folder either through user manipulation in the Finder or through scripts/automation. I don't think I can assume only Finder-originated copying. António - Perfume is the forgiveness that the trampled flower casts upon the heel that crushes it. - ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Notification of file system modification arrives too early?
On May 28, 2010, at 7:57 AM, Antonio Nunes wrote: I expect many of the files will arrive in the watched folder either through user manipulation in the Finder or through scripts/automation. I don't think I can assume only Finder-originated copying. Then there's no way for the system to know when the operation is done. The file being closed would be a very good candidate, but not necessarily... Even more so, the file being open doesn't necessarily mean it's still being copied (Spotlight, for instance). Generally the most reliable way to deal with this is requirements on the process putting the file in the folder. In other words do not copy, and most certainly do not download, a file to the final name/location from where it will be picked up. Copy (or download) to a temporary file, distinguished by name or extension or type or visibility or location, then when the file is complete rename/update/move. If you're only dealing with PDF files, you could check for the presence of the PDF footer at the end of the file. That would work unless some process creates a complete PDF file, then appends pages to it. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
