Re: [zfs-discuss] raidz and compression, difficulties
Joachim Pihl wrote: > On Sun, 27 Jan 2008 19:11:33 +0100, MC <[EMAIL PROTECTED]> wrote: > >>> I didn't expect miracles, but since WinRAR gave 13% compression >> ZFS doesn't compress a block if it can't get a certain amount of return >> on it. >> >> Since the default compression is less effective than RAR, you can bet >> ZFS is seeing much less than 13% return. > > What is the threshold, does anyone know that? It sound like a fair > decision, though. Try to compress on write, increase perrformance on read > by not compressing if there is little space to be gained. Wouldn't slow > down writes compared to not having such a threshold either. Clever! See line 108 of this file: http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/fs/zfs/zio_compress.c#73 -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On Sun, 27 Jan 2008 19:11:33 +0100, MC <[EMAIL PROTECTED]> wrote: >> I didn't expect miracles, but since WinRAR gave 13% compression > > ZFS doesn't compress a block if it can't get a certain amount of return > on it. > > Since the default compression is less effective than RAR, you can bet > ZFS is seeing much less than 13% return. What is the threshold, does anyone know that? It sound like a fair decision, though. Try to compress on write, increase perrformance on read by not compressing if there is little space to be gained. Wouldn't slow down writes compared to not having such a threshold either. Clever! > I expect everything is working properly, and you are just dealing with > well-compressed files :) Raw audio and uncompressed 16bit TIFF files (35mm negative scans) have high content entropy, so they are difficult to compress, that could be the culprit. > You could try setting the zfs compression to a stronger algorithm to see > if anything gets compressed that way. -- Joachim Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On Sun, 27 Jan 2008 19:39:40 +0100, Tim <[EMAIL PROTECTED]> wrote: > Why not create a sample dataset and throw some large .txt files out there > and see what happens? That way you'll know for certain if there's some > bug > you're hitting, or if it's just not applicable to your current dataset. Good idea, but not tonight... It try two things: creating a few dozen GBs worth of "The quick brown fox...", and then create a temporary file system, enable the compression _before_ adding anything to it, then copy the same data to the new file system to see if I can spot any difference. Is there any way to check the compression of a single file in zfs, btw? Would come in handy for this task, and sifting through tons of man files is _not_ my favorite hobby. :-) -- Joachim Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
Why not create a sample dataset and throw some large .txt files out there and see what happens? That way you'll know for certain if there's some bug you're hitting, or if it's just not applicable to your current dataset. On 1/27/08, Joachim Pihl <[EMAIL PROTECTED]> wrote: > > On Sat, 26 Jan 2008 23:33:16 +0100, Toby Thain <[EMAIL PROTECTED]> wrote: > > > On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote: > > >> So far so good, "zfs get all" reports compression to be active. Now for > >> the problem: After adding another 300GB of uncompressed .tif and > >> .bin/.cue > >> (audio CD) files, > > > > I wouldn't expect very much compression on that kind of data. > > I didn't expect miracles, but since WinRAR gave 13% compression on a > representative data set with ZIP compression at the fastest setting, I was > expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting > 5-10% more space on a drive never hurts. > > Could it be that I should specify compression algorithm? Documentation > says it will use lzjb by default, so I did not set it explicitly. > > > -- > Joachim > > Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
> I didn't expect miracles, but since WinRAR gave 13% compression ZFS doesn't compress a block if it can't get a certain amount of return on it. Since the default compression is less effective than RAR, you can bet ZFS is seeing much less than 13% return. I expect everything is working properly, and you are just dealing with well-compressed files :) You could try setting the zfs compression to a stronger algorithm to see if anything gets compressed that way. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On Sat, 26 Jan 2008 23:33:16 +0100, Toby Thain <[EMAIL PROTECTED]> wrote: > On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote: >> So far so good, "zfs get all" reports compression to be active. Now for >> the problem: After adding another 300GB of uncompressed .tif and >> .bin/.cue >> (audio CD) files, > > I wouldn't expect very much compression on that kind of data. I didn't expect miracles, but since WinRAR gave 13% compression on a representative data set with ZIP compression at the fastest setting, I was expecting to see a compression ratio > 1.05 at least, not == 1.00. Getting 5-10% more space on a drive never hurts. Could it be that I should specify compression algorithm? Documentation says it will use lzjb by default, so I did not set it explicitly. -- Joachim Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On Jan 26, 2008, at 3:24 AM, Joachim Pihl wrote:
> So far so good, "zfs get all" reports compression to be active. Now
> for
> the problem: After adding another 300GB of uncompressed .tif
> and .bin/.cue
> (audio CD) files, compression ratio is still at 1.00, indicating
> that no
> compression has taken place.
TIFF files can how their own compression ("compressed TIFF") and many
image editors have this on by default, so you wouldn't know about it
unless you specifically looked for it. I think the TIFF spec specifies
LZW compression for this... but either way, if this is indeed the
case, zfs compression won't help with those
Now, the bin/cue file format specifies no optional compression, so
those bin files should be nothing but a raw image of 16bit PCM
audio... which you should see some (but not great) compression with.
The default lzjb compression scheme in zfs might not be terribly
effective on this type of file data being that it's optimized for
speed rather than compression efficiency. Try turning on gzip
compression in zfs instead and see if things improve.
To make that simple I'd just make a new fs (eg; pool/data/audio) and
then 'zfs set compression=gzip-4 pool/data/audio' and then mv your bin/
cue files there.
/dale
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On 26-Jan-08, at 2:24 AM, Joachim Pihl wrote: > Running SXDE (snv_70) for a file server, and I must admit I'm new to > Solaris and zfs. zfs does not appear to do any compression at all, > here is > what I did to set it up: > > I created a four drive raidz array: > > zpool create pool raidz c0d0 c0d1 c1d0 c1d1 > > then an extra file system on that: > > zfs create -o mountpoint=/export/data pool/data > > After adding a bit of data (200GB or so), i figured that > compression would > in fact be a good idea: > > zfs set compression=on pool/data > > So far so good, "zfs get all" reports compression to be active. Now > for > the problem: After adding another 300GB of uncompressed .tif > and .bin/.cue > (audio CD) files, I wouldn't expect very much compression on that kind of data. --Toby > compression ratio is still at 1.00, indicating that no > compression has taken place. Where did I go wrong? Anything I can > do to > get compression back after the cause for the problem has been > fixed? All > the data has been copied across a network, through Samba - set up > according to instructions found in various Solaris forums. > > -- > Joachim > > Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] raidz and compression, difficulties
On Sat, 26 Jan 2008 15:20:01 +0100, Rob Logan <[EMAIL PROTECTED]> wrote: > it won't go back and compress old data, but if you copy > the files, perhaps into a new folder, they will be > compressed on that copy or move. I can't see why that should help, since the 300GB I added after enabling compression hasn't been compressed at all. The compression is only seemingly enabled, is this a bug with build 70, or is there some misconfiguration at my place? -- Joachim Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] raidz and compression, difficulties
Running SXDE (snv_70) for a file server, and I must admit I'm new to Solaris and zfs. zfs does not appear to do any compression at all, here is what I did to set it up: I created a four drive raidz array: zpool create pool raidz c0d0 c0d1 c1d0 c1d1 then an extra file system on that: zfs create -o mountpoint=/export/data pool/data After adding a bit of data (200GB or so), i figured that compression would in fact be a good idea: zfs set compression=on pool/data So far so good, "zfs get all" reports compression to be active. Now for the problem: After adding another 300GB of uncompressed .tif and .bin/.cue (audio CD) files, compression ratio is still at 1.00, indicating that no compression has taken place. Where did I go wrong? Anything I can do to get compression back after the cause for the problem has been fixed? All the data has been copied across a network, through Samba - set up according to instructions found in various Solaris forums. -- Joachim Hvorfor bruke fremmedord, når det finnes adekvate norske substitutter? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
