Re: NSFileHandle or a better way?
On Jan 30, 2009, at 7:44 AM, Jaime Magiera wrote: I've been using NSFIleHandle for a project that inserts data into a file and synchs it back to disk. Everything went smoothly until the app started getting used for larger files ( 200 megs). First, I ran into the NSFileHandle - NSData 256 megs conundrum. That was solved by, as others suggested on this list in other threads, iterating with readDataOfLength or availableData instead of a single readDataToEndOfFile. That worked well. I've got pools set up to keep the memory down. The problem now is that the reads are still really slow. For example, with a read length of 50 megs, I can only get in 3 reads per second. My apps ends up taking almost a minute to perform all of its functions on a file of 500 megs. Perhaps my overall approach was wrong to start out. What I've been doing is opening a file handle, copying the data after the insertion point to an NSData, truncating the file handle at the insertion point, adding the new data, then adding back the trimmed data. This works fairly well if the insert point is towards the end of the file. However, there are instances where I need to insert a few hundred kb into a the file at a location only a few hundred kb into the file. xxx ^ x The time hit comes from copying the trim data to the NSData. Is there a better way to do this with NSFileHandle? Is there a better way to do this than NSFileHandle? I suspect your bottleneck is the filesystem. To know for sure you could try the raw C calls and see if it speeds up. In any case, instead of doing a grow/shrink on the file, write to a temp file instead then swap them. This way you could also do optimized batch- writes. This alone could speed the process up immensely. Notwithstanding, how do you recover if your app crashes after you truncate the file? HTH, Keary Suska Esoteritech, Inc. Demystifying technology for your home or business ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSFileHandle or a better way?
On Jan 30, 2009, at 11:08 AM, Keary Suska wrote: I suspect your bottleneck is the filesystem. To know for sure you could try the raw C calls and see if it speeds up. In any case, instead of doing a grow/shrink on the file, write to a temp file instead then swap them. This way you could also do optimized batch- writes. This alone could speed the process up immensely. Notwithstanding, how do you recover if your app crashes after you truncate the file? Hi Keary, Thanks for the response. The project did originally start out writing to temp files for these processes. However, I got a complaint from a customer about the disk footprint of temp files. So, I tried going the other route. (I explained the value of tmp files, but they felt the tradeoff wasn't worth it). Now, it seems the original path was the best regardless. I shall return to that method. thanks for the advice, Jaime Magiera Sensory Research http://www.sensoryresearch.net ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: NSFileHandle or a better way?
On 30 Jan 09, at 06:44, Jaime Magiera wrote: I've been using NSFIleHandle for a project that inserts data into a file and synchs it back to disk. ... My apps ends up taking almost a minute to perform all of its functions on a file of 500 megs. Perhaps my overall approach was wrong to start out. What I've been doing is opening a file handle, copying the data after the insertion point to an NSData, truncating the file handle at the insertion point, adding the new data, then adding back the trimmed data. This works fairly well if the insert point is towards the end of the file. However, there are instances where I need to insert a few hundred kb into a the file at a location only a few hundred kb into the file. xxx ^ x The time hit comes from copying the trim data to the NSData. Is there a better way to do this with NSFileHandle? Is there a better way to do this than NSFileHandle? Not without changing the structure of your file. There's no way to shift the contents of a file in the way you're looking for - UNIX file systems aren't structured in a way that permits that. (In fact, I'm not aware of any that are.) You'll need to either rethink how you're storing data, or cope with slow saves. Incidentally, the way you're doing saving is unsafe. If your application crashes in the middle of a save operation, the file may end up truncated. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: http://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
