Well, then you could have more "logical space" than "physical space"
Reconsidering my own question again, it seems to me that the question of space management is probably more fundamental than I had initially thought, and I assume members of the core team will have thought through much of it.
I will try to share my thoughts and I would very much appreciate any corrections or additional explanations.
For dedup, my understanding at this point is that, first of all, every reference to dedup'ed data must be accounted to the respective dataset.
Obviously, a decision has been made to account that space as "used", rather than "referenced". I am trying to understand, why.
At first sight, referring to the definition of "used" space as being unique to the respective dataset, it would seem natural to account all de-duped space as "referenced". But this could lead to much space never being accounted as "used" anywhere (but for the pool). This would differs from the observed behavior from non-deduped datasets, where, to my understanding, all "referred" space is "used" by some other dataset. Despite being a little counter-intuitive, first I found this simple solution quite attractive, because it wouldn't alter the semantics of used vs. referenced space (under the assumption that my understanding is correct).
My understanding from Eric's explanation is that it has been decided to go an alternative route and account all de-duped space as "used" to all datasets referencing it because, in contrast to snapshots/clones, it is impossible (?) to differentiate between used and referred space for de-dup. Also, at first sight, this seems to be a way to keep the current semantics for (ref)reservations.
But while without de-dup, all the usedsnap and usedds values should roughly sum up to the pool used space, they can't with this concept - which is why I thought a solution could be to compensate for multiply accounted "used" space by artificially increasing the pool size.
Instead, from the examples given here, what seems to have been implemented with de-dup is to simply maintain space statistics for the pool on the basis of actually used space.
While one find it counter-intuitive that the used sizes of all datasets/snapshots will exceed the pool used size with de-dedup, if my understanding is correct, this design seems to be consistent.
I am very interested in the reasons why this particular approach has been chosen and why others have been dropped.
Now to the more general question: If all datasets of a pool contained the same data and got de-duped, the sums of their "used" space still seems to be limited by the "locical" pool size, as we've seen in examples given by Jürgen and others and, to get a benefit of de-dup, this implementation obviously needs to be changed.
But: Isn't there an implicit expectation for a space guarantee associated with a dataset? In other words, if a dataset has 1GB of data, isn't it natural to expect to be able to overwrite that space with other data? One might want to define space guarantees (like with (ref)reservation), but I don't see how those should work with the currently implemented concept.
Do we need something like a de-dup-reservation, which is substracted from the pool free space?
Thank you for reading, Nils _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
