Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
I wouldn't blame the rsync team for not wanting to maintain it, it's a pretty narrow-scope patch affecting only one OS. I'm pretty motivated to keep it up, though, so I'll repost my patches to this list when I update them. I'll probably get it updated to 3.1.0 in the next month or so. Mike On Nov 1, 2009, at 7:57 PM, Tony wrote: Mike, thanks for the patch. Will this patch be maintained in rsync- patches-3.0.6.tar.gz ? On Oct 28, 2009, at 1:20 AM, Mike Bombich wrote: HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved. A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40). When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data. It should also pass a special flag to the xattr calls to expose the decmpfs xattrs. I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch. I did that this evening and attached it below. These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches. It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required). Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction. Mike On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote: On Tue, 2009-10-27 at 23:38 -0400, Tony wrote: When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed. I examined the destination file as described in ars technica (with ls and xattr from a 10.5 Leopard boot), and the compressed data is moved from the resource fork to the data fork, and the extended attributes '@' are removed from the file. As far as I know, only ditto in 10.6 can handle HFS+ File Compression. (I even tested a 'clone' with disk utility (file copy, not block), and it also failed (block copy, of course works). Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
Mike, thanks for the patch. Will this patch be maintained in rsync- patches-3.0.6.tar.gz ? On Oct 28, 2009, at 1:20 AM, Mike Bombich wrote: HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved. A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40). When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data. It should also pass a special flag to the xattr calls to expose the decmpfs xattrs. I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch. I did that this evening and attached it below. These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches. It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required). Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction. Mike On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote: On Tue, 2009-10-27 at 23:38 -0400, Tony wrote: When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed. I examined the destination file as described in ars technica (with ls and xattr from a 10.5 Leopard boot), and the compressed data is moved from the resource fork to the data fork, and the extended attributes '@' are removed from the file. As far as I know, only ditto in 10.6 can handle HFS+ File Compression. (I even tested a 'clone' with disk utility (file copy, not block), and it also failed (block copy, of course works). Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
HFS compression can be preserved as long as the relevant xattr(s) and flags on those files are preserved. A compressed file has the compressed data in a hidden xattr (com.apple.decmpfs if < 4Kb, com.apple.ResourceFork if more), and has the UF_COMPRESSED flag set (decimal 40). When rsync encounters a file like this, it should ignore the data fork of the file, which will appear to contain normal, uncompressed data. It should also pass a special flag to the xattr calls to expose the decmpfs xattrs.I've already implemented this in rsync (3.0.6), I just hadn't taken the time to craft the HFS-compression-specific changes into a patch. I did that this evening and attached it below. These are changes against the 3.0.6 base, plus the crtimes, fileflags, and backup-dir-dels patches. It should work, at minimum, against the 3.0.6 base plus the fileflags patch (that patch is required).Let me know if it doesn't work for you, it's entirely possible that I overlooked something in the extraction.Mike rsync_3.0.6-hfs-compression_20091027.diff Description: Binary data On Oct 27, 2009, at 11:08 PM, Matt McCutchen wrote:On Tue, 2009-10-27 at 23:38 -0400, Tony wrote:When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed.I examined the destination file as described in ars technica (with ls and xattr from a 10.5 Leopard boot), and the compressed data is moved from the resource fork to the data fork, and the extended attributes '@' are removed from the file.As far as I know, only ditto in 10.6 can handle HFS+ File Compression. (I even tested a 'clone' with disk utility (file copy, not block), and it also failed (block copy, of course works).Rsync is just reading and writing files via the filesystem API; it hasno access to any of the flags or xattrs used to implement thecompression.I guess the filesystem doesn't compress new files by default. If it hadan API to request compression, rsync could use that API when writing thedestination files. Unfortunately, the API ditto is using appears to beprivate to Apple. See the post from brkirch beginning "The first thingthat I tried to do" on this page:http://www.macosxhints.com/article.php?story=20090902223042255So anyone interested in making rsync compress the destination fileswould probably have to copy the relevant code from afsctool. This couldbe shared as a patch; I feel quite sure it would not be adopted in themain version of rsync.-- Matt-- Please use reply-all for most replies to avoid omitting the mailing list.To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsyncBefore posting, read: http://www.catb.org/~esr/faqs/smart-questions.html-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
Thanks. The C code that brkirch provides takes care of a lot of the work, so hopefully someone will be able to provide a patch (Its been over 15 years since I did any C programing, so unfortunately I won't be able to contribute) On Oct 28, 2009, at 12:08 AM, Matt McCutchen wrote: Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. On Tue, 2009-10-27 at 23:38 -0400, Tony wrote: When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed. I examined the destination file as described in ars technica (with ls and xattr from a 10.5 Leopard boot), and the compressed data is moved from the resource fork to the data fork, and the extended attributes '@' are removed from the file. As far as I know, only ditto in 10.6 can handle HFS+ File Compression. (I even tested a 'clone' with disk utility (file copy, not block), and it also failed (block copy, of course works). Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
When rsync 3.0.6 copies files with HFS+ File Compression, the new extended attribute decmpfs is not preserved, and the UF_COMPRESSED flag is not set on the destination and the destination file is not compressed. I examined the destination file as described in ars technica (with ls and xattr from a 10.5 Leopard boot), and the compressed data is moved from the resource fork to the data fork, and the extended attributes '@' are removed from the file. As far as I know, only ditto in 10.6 can handle HFS+ File Compression. (I even tested a 'clone' with disk utility (file copy, not block), and it also failed (block copy, of course works). On Oct 27, 2009, at 7:39 PM, Matt McCutchen wrote: What kind of special treatment from rsync were you expecting? I read http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3 , and as far as I can tell, the compression is handled entirely by the filesystem with no intervention from applications needed. -- Matt On Tue, 2009-10-27 at 19:31 -0400, Tony wrote: Are there any patches (or planned updates) to rsync v3.0.6 to handle the HFS+ File Compression that Apple introduced with Snow Leopard? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
On Tue, 2009-10-27 at 23:38 -0400, Tony wrote: > When rsync 3.0.6 copies files with HFS+ File Compression, the new > extended attribute decmpfs is not preserved, and the UF_COMPRESSED > flag is not set on the destination and the destination file is not > compressed. > > I examined the destination file as described in ars technica (with ls > and xattr from a 10.5 Leopard boot), and the compressed data is moved > from the resource fork to the data fork, and the extended attributes > '@' are removed from the file. > > As far as I know, only ditto in 10.6 can handle HFS+ File > Compression. (I even tested a 'clone' with disk utility (file copy, > not block), and it also failed (block copy, of course works). Rsync is just reading and writing files via the filesystem API; it has no access to any of the flags or xattrs used to implement the compression. I guess the filesystem doesn't compress new files by default. If it had an API to request compression, rsync could use that API when writing the destination files. Unfortunately, the API ditto is using appears to be private to Apple. See the post from brkirch beginning "The first thing that I tried to do" on this page: http://www.macosxhints.com/article.php?story=20090902223042255 So anyone interested in making rsync compress the destination files would probably have to copy the relevant code from afsctool. This could be shared as a patch; I feel quite sure it would not be adopted in the main version of rsync. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OS X 10.6 (Snow Leopard) HFS+ File Compression
On Tue, 2009-10-27 at 19:31 -0400, Tony wrote: > Are there any patches (or planned updates) to rsync v3.0.6 to handle > the HFS+ File Compression that Apple introduced with Snow Leopard? What kind of special treatment from rsync were you expecting? I read http://arstechnica.com/apple/reviews/2009/08/mac-os-x-10-6.ars/3 , and as far as I can tell, the compression is handled entirely by the filesystem with no intervention from applications needed. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
