[zfs-discuss] ZFS default compression and file size limit?
I have a 13GB text file. I turned ZFS compression on with zfs set compression=on mypool. When i copy the 13GB file into another file, it does not get compressed (checking via du -sh). However if i set compression=gzip, then the file gets compressed. Is there a limit on file size with the default compression algorithm? I did experiment with a much smaller file of 0.5GB with the default compression and it did get compressed. I am using S10 U8 x86/64. Regards, -- Wajih Ahmed Principal Field Technologist 877.274.6589 / x40572 Skype: wajih_ahmed ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS default compression and file size limit?
On 20/01/2010 13:39, Wajih Ahmed wrote: I have a 13GB text file. I turned ZFS compression on with zfs set compression=on mypool. When i copy the 13GB file into another file, it does not get compressed (checking via du -sh). However if i set compression=gzip, then the file gets compressed. Is there a limit on file size with the default compression algorithm? I did experiment with a much smaller file of 0.5GB with the default compression and it did get compressed. if a given block is not gaining more than 12.5% from a compression then it will not be stored as compressed. It might be that with a default compression algorithm (lzjb) you are gaining less than 12.5% while when using gzip you are getting more therefore blocks end up being compressed. -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS default compression and file size limit?
I have a 13GB text file. I turned ZFS compression on with zfs set compression=on mypool. When i copy the 13GB file into another file, it does not get compressed (checking via du -sh). However if i set compression=gzip, then the file gets compressed. Is there a limit on file size with the default compression algorithm? I did experiment with a much smaller file of 0.5GB with the default compression and it did get compressed. I am using S10 U8. Regards, -- Wajih Ahmed Principal Field Technologist 877.274.6589 / x40572 Skype: wajih_ahmed ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS default compression and file size limit?
Mike, Thank you for your quick response... Is there a way for me to test the compression from the command line to see if lzjb is giving me more or less than the 12.5% mark? I guess it will depend if there is a lzjb command line utility. I am just a little surprised because gzip-6 is able to compress it to 4.4GB from 14GB (and gzip-1 4.8GB) and from what i read lzjb should be giving me better an 12.5% compression. For example the *compress* command (which i think uses LZO, a slight different variant of Lempel-Ziv) manges to reduce it to 8.0GB. That is a 57% ratio. Regards, -- Wajih Ahmed Principal Field Technologist 877.274.6589 / x40572 Skype: wajih_ahmed Robert Milkowski wrote: On 20/01/2010 13:39, Wajih Ahmed wrote: I have a 13GB text file. I turned ZFS compression on with zfs set compression=on mypool. When i copy the 13GB file into another file, it does not get compressed (checking via du -sh). However if i set compression=gzip, then the file gets compressed. Is there a limit on file size with the default compression algorithm? I did experiment with a much smaller file of 0.5GB with the default compression and it did get compressed. if a given block is not gaining more than 12.5% from a compression then it will not be stored as compressed. It might be that with a default compression algorithm (lzjb) you are gaining less than 12.5% while when using gzip you are getting more therefore blocks end up being compressed. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS default compression and file size limit?
On Wed, Jan 20, 2010 at 12:42:35PM -0500, Wajih Ahmed wrote: Mike, Thank you for your quick response... Is there a way for me to test the compression from the command line to see if lzjb is giving me more or less than the 12.5% mark? I guess it will depend if there is a lzjb command line utility. I am just a little surprised because gzip-6 is able to compress it to 4.4GB from 14GB (and gzip-1 4.8GB) and from what i read lzjb should be giving me better an 12.5% compression. For example the *compress* command (which i think uses LZO, a slight different variant of Lempel-Ziv) manges to reduce it to 8.0GB. That is a 57% ratio. That's over the whole file as a single compression stream. ZFS has to compress each block (128k or maybe less) independently. This can't do as well. -- Dan. pgpdvR9z5t17a.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
