Re: [zfs-discuss] ZFS dedup issue
Hello all Sorry for bumping an old thread, but now that snv_128 is due to appear as a public DVD download, I wonder: has this fix for zfs-accounting and other issues with zfs dedup been integrated into build 128? We have a fileserver which is likely to have much redundant data and we'd like to clean up its space with zfs-deduping (even if that takes copying files over to a temp dir and back - so their common blocks are noticed by the code). Will build 128 be ready for the task - and increase our server's available space after deduping - or should we better wait for another one? In general, were there any stability issues with snv_128 during internal/BFU testing? TIA, //Jim -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
Hi Jim, Nevada build 128 had some problems so will not be released. The dedup space fixes should be available in build 129. Thanks, Cindy On 12/02/09 02:37, Jim Klimov wrote: Hello all Sorry for bumping an old thread, but now that snv_128 is due to appear as a public DVD download, I wonder: has this fix for zfs-accounting and other issues with zfs dedup been integrated into build 128? We have a fileserver which is likely to have much redundant data and we'd like to clean up its space with zfs-deduping (even if that takes copying files over to a temp dir and back - so their common blocks are noticed by the code). Will build 128 be ready for the task - and increase our server's available space after deduping - or should we better wait for another one? In general, were there any stability issues with snv_128 during internal/BFU testing? TIA, //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
Hey Cindy! Any idea of when we might see 129? (an approximation only). I ask the question because I'm pulling budget funds to build a filer, but it may not be in service until mid-January. Would it be reasonable to say that we might see 129 by then, or are we looking at summer...or even beyond? I don't see that there's a wrong answer: here necessarily, :) :) :) I'll go with what's out, but dedup is a big one and a feature that made me commit to this project. -Colin On Wed, Dec 2, 2009 at 17:06, Cindy Swearingen cindy.swearin...@sun.comwrote: Hi Jim, Nevada build 128 had some problems so will not be released. The dedup space fixes should be available in build 129. Thanks, Cindy On 12/02/09 02:37, Jim Klimov wrote: Hello all Sorry for bumping an old thread, but now that snv_128 is due to appear as a public DVD download, I wonder: has this fix for zfs-accounting and other issues with zfs dedup been integrated into build 128? We have a fileserver which is likely to have much redundant data and we'd like to clean up its space with zfs-deduping (even if that takes copying files over to a temp dir and back - so their common blocks are noticed by the code). Will build 128 be ready for the task - and increase our server's available space after deduping - or should we better wait for another one? In general, were there any stability issues with snv_128 during internal/BFU testing? TIA, //Jim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
Colin Raven wrote: Hey Cindy! Any idea of when we might see 129? (an approximation only). I ask the question because I'm pulling budget funds to build a filer, but it may not be in service until mid-January. Would it be reasonable to say that we might see 129 by then, or are we looking at summer...or even beyond? I don't see that there's a wrong answer: here necessarily, :) :) :) I'll go with what's out, but dedup is a big one and a feature that made me commit to this project. The unstable and experimental Sun builds typically lag about 2 weeks behind the cut of the hg tag. (Holidays and respins can derail that of course.) The stable releases I have no clue about. Depending on the level of adventure osunix in our next release may be interesting to you. Feel free to email me off list or say hi on irc #osunix irc.freenode.net Thanks ./C ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
So.. it seems that data is deduplicated, zpool has 54.1G of free space, but I can use only 40M. It's x86, ONNV revision 10924, debug build, bfu'ed from b125. I think I'm observing the same (with changeset 10936) ... I created a 2GB file, and a tank zpool on top of that file, with compression and dedup enabled: mkfile 2g /var/tmp/tank.img zpool create tank /var/tmp/tank.img zfs set dedup=on tank zfs set compression=on tank Now I tried to create four zfs filesystems, and filled them by pulling and updating the same set of onnv sources from mercurial. One copy needs ~ 800MB of disk space uncompressed, or ~ 520MB compressed. During the 4th hg update: hg update abort: No space left on device: /tank/snv_128_yy/usr/src/lib/libast/sparcv9/src/lib/libast/FEATURE/common zpool list tank NAME SIZE USED AVAILCAP DEDUP HEALTH ALTROOT tank 1,98G 720M 1,28G35% 3.70x ONLINE - zfs list -r tank NAME USED AVAIL REFER MOUNTPOINT tank 1,95G 026K /tank tank/snv_128 529M 0 529M /tank/snv_128 tank/snv_128_jk 530M 0 530M /tank/snv_128_jk tank/snv_128_xx 530M 0 530M /tank/snv_128_xx tank/snv_128_yy 368M 0 368M /tank/snv_128_yy -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
On Nov 3, 2009, at 6:01 AM, Jürgen Keil wrote: I think I'm observing the same (with changeset 10936) ... # mkfile 2g /var/tmp/tank.img # zpool create tank /var/tmp/tank.img # zfs set dedup=on tank # zfs create tank/foobar This has to do with the fact that dedup space accounting is charged to all filesystems, regardless of whether blocks are deduped. To do otherwise is impossible, as there is no true owner of a block, and the fact that it may or may not be deduped is often beyond the control of a single filesystem. This has some interesting pathologies as the pool gets full. Namely, that ZFS will artificially enforce a limit on the logical size of the pool based on non-deduped data. This is obviously something that should be addressed. - Eric dd if=/dev/urandom of=/tank/foobar/file1 bs=1024k count=512 512+0 records in 512+0 records out cp /tank/foobar/file1 /tank/foobar/file2 cp /tank/foobar/file1 /tank/foobar/file3 cp /tank/foobar/file1 /tank/foobar/file4 /tank/foobar/file4: No space left on device zfs list -r tank NAME USED AVAIL REFER MOUNTPOINT tank 1.95G 022K /tank tank/foobar 1.95G 0 1.95G /tank/foobar zpool list tank NAME SIZE USED AVAILCAP DEDUP HEALTH ALTROOT tank 1.98G 515M 1.48G25% 3.90x ONLINE - -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -- Eric Schrock, Fishworkshttp://blogs.sun.com/eschrock ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
Cyril Plisko wrote: I think I'm observing the same (with changeset 10936) ... # mkfile 2g /var/tmp/tank.img # zpool create tank /var/tmp/tank.img # zfs set dedup=on tank # zfs create tank/foobar This has to do with the fact that dedup space accounting is charged to all filesystems, regardless of whether blocks are deduped. To do otherwise is impossible, as there is no true owner of a block, and the fact that it may or may not be deduped is often beyond the control of a single filesystem. This has some interesting pathologies as the pool gets full. Namely, that ZFS will artificially enforce a limit on the logical size of the pool based on non-deduped data. This is obviously something that should be addressed. Eric, Many people (me included) perceive deduplication as a mean to save disk space and allow more data to be squeezed into a storage. What you are saying is that effectively ZFS dedup does a wonderful job in detecting duplicate blocks and goes into all the trouble of removing an extra copies and keep accounting of everything. However, when it comes to letting me use the freed space I will be plainly denied to do so. If that so, what would be the reason to use ZFS deduplication at all ? c'mon it is obviously a bug and not a design feature. (it is I hope/think that is the case) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
I'm fairly new to all this and I think that is the intended behavior. Also from my limited understanding I believe dedup behavior it would significantly cut down on access times. For the most part though this is such new code that I would wait abit to see where they take it. On Tue, Nov 3, 2009 at 3:24 PM, Cyril Plisko cyril.pli...@mountall.comwrote: I think I'm observing the same (with changeset 10936) ... # mkfile 2g /var/tmp/tank.img # zpool create tank /var/tmp/tank.img # zfs set dedup=on tank # zfs create tank/foobar This has to do with the fact that dedup space accounting is charged to all filesystems, regardless of whether blocks are deduped. To do otherwise is impossible, as there is no true owner of a block, and the fact that it may or may not be deduped is often beyond the control of a single filesystem. This has some interesting pathologies as the pool gets full. Namely, that ZFS will artificially enforce a limit on the logical size of the pool based on non-deduped data. This is obviously something that should be addressed. Eric, Many people (me included) perceive deduplication as a mean to save disk space and allow more data to be squeezed into a storage. What you are saying is that effectively ZFS dedup does a wonderful job in detecting duplicate blocks and goes into all the trouble of removing an extra copies and keep accounting of everything. However, when it comes to letting me use the freed space I will be plainly denied to do so. If that so, what would be the reason to use ZFS deduplication at all ? -- Regards, Cyril ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
I think I'm observing the same (with changeset 10936) ... # mkfile 2g /var/tmp/tank.img # zpool create tank /var/tmp/tank.img # zfs set dedup=on tank # zfs create tank/foobar This has to do with the fact that dedup space accounting is charged to all filesystems, regardless of whether blocks are deduped. To do otherwise is impossible, as there is no true owner of a block, and the fact that it may or may not be deduped is often beyond the control of a single filesystem. This has some interesting pathologies as the pool gets full. Namely, that ZFS will artificially enforce a limit on the logical size of the pool based on non-deduped data. This is obviously something that should be addressed. Eric, Many people (me included) perceive deduplication as a mean to save disk space and allow more data to be squeezed into a storage. What you are saying is that effectively ZFS dedup does a wonderful job in detecting duplicate blocks and goes into all the trouble of removing an extra copies and keep accounting of everything. However, when it comes to letting me use the freed space I will be plainly denied to do so. If that so, what would be the reason to use ZFS deduplication at all ? -- Regards, Cyril ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS dedup issue
Eric Schrock wrote: On Nov 3, 2009, at 12:24 PM, Cyril Plisko wrote: I think I'm observing the same (with changeset 10936) ... # mkfile 2g /var/tmp/tank.img # zpool create tank /var/tmp/tank.img # zfs set dedup=on tank # zfs create tank/foobar This has to do with the fact that dedup space accounting is charged to all filesystems, regardless of whether blocks are deduped. To do otherwise is impossible, as there is no true owner of a block, and the fact that it may or may not be deduped is often beyond the control of a single filesystem. This has some interesting pathologies as the pool gets full. Namely, that ZFS will artificially enforce a limit on the logical size of the pool based on non-deduped data. This is obviously something that should be addressed. Eric, Many people (me included) perceive deduplication as a mean to save disk space and allow more data to be squeezed into a storage. What you are saying is that effectively ZFS dedup does a wonderful job in detecting duplicate blocks and goes into all the trouble of removing an extra copies and keep accounting of everything. However, when it comes to letting me use the freed space I will be plainly denied to do so. If that so, what would be the reason to use ZFS deduplication at all ? Please read my response before you respond. What do you think this is obviously something that should be addressed means? There is already a CR filed and the ZFS team is working on it. We have a fix for this and it should be available in a couple of days. - George - Eric -- Regards, Cyril -- Eric Schrock, Fishworks http://blogs.sun.com/eschrock ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
