[zfs-discuss] Re: Proposal: multiple copies of user
> On 9/15/06, can you guess? <[EMAIL PROTECTED]> > wrote: ... file-level, however, is really pushing > it. You might end > up with an administrative nightmare deciphering which > files have how > many copies.\ I'm not sure what you mean: the level of redundancy would be a per-file attribute that could be examined, and would be normally just be defaulted to a common value. ... > > It would be interesting to know whether that would > still be your experience in environments that > regularly scrub active data as ZFS does (assuming > that said experience was accumulated in environments > that don't). The theory behind scrubbing is that all > data areas will be hit often enough that they won't > have time to deteriorate (gradually) to the point > where they can't be read at all, and early > deterioration encountered during the scrub pass (or > other access) in which they have only begun to become > difficult to read will result in immediate > revectoring (by the disk or, if not, by the file > system) to healthier locations. > > Scrubbing exercises the disk area to prevent bit-rot. > I do not think > FS's scrubbing changes the failure mode of the raw > devices. It doesn't change the failure rate (if anything, it might accelerate it marginally due to the extra disk activity), but it *does* change, potentially radically, the frequency with which sectors containing user data become unreadable - because it allows them to be detected *before* that happens such that the data can be moved to a good sector (often by the disk itself, else by higher-level software) and the failing sector marked bad. OTOH, I > really have no such experience to speak of *fingers > crossed*. I > failed to locate the code where the relocation of > files happens but > assume that copies would make this process more > reliable. Sort of: while they don't make any difference when you catch a failing sector while it's still readable, they certainly help if you only catch it after it's become unreadable (or has been 'silently' corrupted). > > > Since ZFS-style scrubbing detects even > otherwise-indetectible 'silent corruption' missed by > the disk's own ECC mechanisms, that lower-probability > event is also covered (though my impression is that > the probability of even a single such sector may be > significantly lower than that of whole-disk failure, > especially in laptop environments). > > I do not any data to support nor dismiss that. Quite a few years ago Seagate still published such data, but of course I didn't copy it down (because it was 'always available' when I wanted it - as I said, it was quite a while ago and I was not nearly as well-acquainted with the volatility of Internet data as I would subsequently become). But to the best of my recollection their enterprise disks at that time were specced to have no worse than 1 uncorrectable error for every petabit read and no worse than 1 undetected error for every exabit read. A fairly recent paper by people who still have access to such data suggests that the frequency of uncorrectable errors in enterprise drives is still about the same, but that the frequency of undetected errors may have increased markedly (to perhaps once in every 10 petabits read) - possibly a result of ever-increasing on-disk bit densities and the more aggressive error correction required to handle them (perhaps this is part of the reason they don't make error rates public any more...). They claim that SATA drives have error rates around 10x that of enterprise drives (or an undetected error rate of around once per petabit). Figure out a laptop drive's average data rate and that gives you a mean time to encountering undetected corruption. Compare that to the drive's in-use MTBF rating and there you go! If I haven't dropped a decimal place or three doing this in my head, then even if laptop drives have nominal MTBFs equal to desktop SATA drives it looks as if it would take an average data rate of 60 - 70 KB/sec (24/7, year-in, year-out) for the likelihood of an undetected error to be comparable in likelihood to a whole-disk failure: that's certainly nothing much for a fairly well-loaded server in constant (or even just 40 hour/week) use, but for a laptop?. - bill This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user
>By the way, is there a way to view just the responses that have accumulated in >this forum since I >last visited - or just those I've never looked at before? Not through the web interface itself, as far as I can tell, but there's an RSS feed of messages that might do the trick. Unfortunately it points to the whole thread, rather than the individual messages. http://opensolaris.org/jive/rss/rssmessages.jspa?forumID=80 > it seems downright silly to gate ZFS facilities > on the basis of two-decade-old network file access technology: sure, > it's important to be able to *access* ZFS files using NFS, but does > anyone really care if NFS can't express the full range of ZFS features - Personally, I don't think it's critical. After all, you can't create a snapshot via NFS either, but we have snapshots. Concepts such as administrative ID, inherited directory characteristics, etc. have had great success in file systems such as IBM's GPFS and Sun's QFS, as well as on NetApp's systems. For that matter, quotas aren't really in NFSv3, but nobody seems to mind that UFS implements them. Anton This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Proposal: multiple copies of user
On September 15, 2006 3:49:14 PM -0700 "can you guess?" <[EMAIL PROTECTED]> wrote: (I looked at my email before checking here, so I'll just cut-and-paste the email response in here rather than send it. By the way, is there a way to view just the responses that have accumulated in this forum since I last visited - or just those I've never looked at before?) subscribe via email instead of reading it as a forum ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user
(I looked at my email before checking here, so I'll just cut-and-paste the email response in here rather than send it. By the way, is there a way to view just the responses that have accumulated in this forum since I last visited - or just those I've never looked at before?) Bill Moore wrote: > On Fri, Sep 15, 2006 at 01:23:31AM -0700, can you guess? wrote: >> Implementing it at the directory and file levels would be even more >> flexible: redundancy strategy would no longer be tightly tied to path >> location, but directories and files could themselves still inherit >> defaults from the filesystem and pool when appropriate (but could be >> individually handled when desirable). > > The problem boils down to not having a way to express your intent that > works over NFS (where you're basically limited by POSIX) that you can > use from any platform (esp. ones where ZFS isn't installed). If you > have some ideas, this is something we'd love to hear about. Well, one idea is that it seems downright silly to gate ZFS facilities on the basis of two-decade-old network file access technology: sure, it's important to be able to *access* ZFS files using NFS, but does anyone really care if NFS can't express the full range of ZFS features - at least to the degree that they think such features should be suppressed as a result (rather than made available to local users plus any remote users employing a possibly future mechanism that *can* support them)? That being said, you could always adopt the ReiserFS approach of allowing access to file/directory metadata via extended path specifications in environments like NFS where richer forms of interaction aren't available: yes, it may feel a bit kludgey, but it gets the job done. And, of course, even if you did nothing to help NFS its users would still benefit from inheriting whatever arbitrarily fine-grained redundancy levels had been established via more comprehensive means: they just wouldn't be able to tweak redundancy levels themselves (any more, or any less, than they can do so today). > >> I've never understood why redundancy was a pool characteristic in ZFS >> - and the addition of 'ditto blocks' and now this new proposal (both >> of which introduce completely new forms of redundancy to compensate >> for the fact that pool-level redundancy doesn't satisfy some needs) >> just makes me more skeptical about it. > > We have thought long and hard about this problem and even know how to > implement it (the name we've been using is Metaslab Grids, which isn't > terribly descriptive, or as Matt put it "a bag o' disks"). Yes, 'a bag o' disks' - used intelligently at a higher level - is pretty much what I had in mind. There are > two main problems with it, though. One is failures. The problem is > that you want the set of disks implementing redundancy (mirror, RAID-Z, > etc.) to be spread across fault domains (controller, cable, fans, power > supplies, geographic sites) as much as possible. There is no generic > mechanism to obtain this information and act upon it. We could ask the > administrator to supply it somehow, but such a description takes effort, > is not easy, and prone to error. That's why we have the model right now > where the administrator specifies how they want the disks spread out > across fault groups (vdevs). Without having looked at the code I may be missing something here. Even with your current implementation, if there's indeed no automated way to obtain such information the administrator has to exercise manual control over disk groupings if they're going to attain higher availability by avoiding other single points of failure instead of just guard against unrecoverable data loss from disk failure. Once that information has been made available to the system, letting it make use of it at a higher level rather than just aggregating entire physical disks should not entail additional administrator effort. I admit that I haven't considered the problem in great detail, since my bias is toward solutions that employ redundant arrays of inexpensive nodes to scale up rather than a small number of very large nodes (in part because a single large node itself can often be a single point of failure even if many of its subsystems carefully avoid being so in the manner that you suggest). Each such small node has a relatively low disk count and little or no internal redundancy, and thus comprises its own little fault-containment environment, avoiding most such issues; as a plus, such node sizes mesh well with the bandwidth available from very inexpensive Gigabit Ethernet interconnects and switches (even when streaming data sequentially, such as video on demand) and allow fine-grained incremental system scaling (by the time faster interconnects become inexpensive, disk bandwidth should have increased enough that such a balance will still be fairly good). Still, if you can group whole disks intelligently in a la
[zfs-discuss] Re: Proposal: multiple copies of user data
Is this true for single-sector, vs. single-ZFS-block, errors? (Yes, it's pathological and probably nobody really cares.) I didn't see anything in the code which falls back on single-sector reads. (It's slightly annoying that the interface to the block device drivers loses the SCSI error status, which tells you the first sector which was bad.) This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Proposal: multiple copies of user
Celso wrote: a couple of points One could make the argument that the feature could cause enough confusion to not warrant its inclusion. If I'm a typical user and I write a file to the filesystem where the admin set three copies but didn't tell me it might throw me into a tizzy trying to figure out why my quota is 3X where I expect it to be. I don't think anybody is saying it is going to be the default setup. If someone is not comfortable with a feature, surely they can choose to ignore it. An admin can use actual mirroring, raidz etc, and carry on as before. There are many potentially confusing features of almost any computer system. Computers are complex things. I admin a couple of schools with a total of about 2000 kids. I really doubt that any of them would have a problem understanding it. More importantly, is an institution utilizing quotas really the main market for this feature. It seems to me that it is clearly aimed at people in control of their own machines (even though I can see uses for this in pretty much any environment). I doubt anyone capable of installing and running Solaris on their laptop would be confused by this issue. Its not the smart people I would be worried about. It's the ones where you would get into endless loops of conversation around "But I only wrote 1MB how come it says 2MB?" that worry me. Especially, when it impacts a lot of user level tools and could be a surprise if set by a BOFH type. That said I was worried about that type of effect when the change itself seemed to have low value. However, you and Richard have pointed to at least one example where this would be useful at the file level Given a situation, where you: a) have a laptop or home computer which you have important data on. b) for whatever reason, you can't add another disk to utilize mirroring (and you are betweenbackups) this seems to me to be a very valid solution. ... and though I see that as a valid solution to the issue does it really cover enough ground to warrant inclusion of this feature given some of the other issues that have been brought up? In the above case I think people would me more concerned with the entire system going down, a drive crashing, etc. then the possibility of a checksum error or data corruption requiring the lookup on a ditto block if one exists. In that case they would create a copy on an independent system, like a USB disk, some sort of archiving media, like a CD-R, or even place a copy on a remote system, to maintain the data in case of a failure. Hell, I've been known to do all three to meet my own paranoia level. IMHO, It's more ammo to include the feature but I'm not sure its enough. Perhaps Richard's late breaking data concerning drive failures will add some more weight? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user
a couple of points > One could make the argument that the feature could > cause enough > confusion to not warrant its inclusion. If I'm a > typical user and I > write a file to the filesystem where the admin set > three copies but > didn't tell me it might throw me into a tizzy trying > to figure out why > my quota is 3X where I expect it to be. > I don't think anybody is saying it is going to be the default setup. If someone is not comfortable with a feature, surely they can choose to ignore it. An admin can use actual mirroring, raidz etc, and carry on as before. There are many potentially confusing features of almost any computer system. Computers are complex things. I admin a couple of schools with a total of about 2000 kids. I really doubt that any of them would have a problem understanding it. More importantly, is an institution utilizing quotas really the main market for this feature. It seems to me that it is clearly aimed at people in control of their own machines (even though I can see uses for this in pretty much any environment). I doubt anyone capable of installing and running Solaris on their laptop would be confused by this issue. I don't think anyone is saying that ditto blocks are a complete, never-lose-data solution. Sure the whole disk can (and probably will) die on you. If you partitioned the disk, mirrored it, and it died, you would still be in trouble. If the disk doesn't die, but for whatever reason, you get silent data corruption, the checksums pick up the problem, and the ditto blocks allow recovery. Given a situation, where you: a) have a laptop or home computer which you have important data on. b) for whatever reason, you can't add another disk to utilize mirroring (and you are between backups) this seems to me to be a very valid solution. Especially as has already been said, it takes very little to implement, and doesn't hinder anything else within zfs. I think that people can benefit from this. > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss > This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user data
> Matthew Ahrens wrote: > > Here is a proposal for a new 'copies' property > which would allow > > different levels of replication for different > filesystems. > > Thanks everyone for your input. > > The problem that this feature attempts to address is > when you have some > data that is more important (and thus needs a higher > level of > redundancy) than other data. Of course in some > situations you can use > multiple pools, but that is antithetical to ZFS's > pooled storage model. > (You have to divide up your storage, you'll end up > with stranded > torage and bandwidth, etc.) > > Given the overwhelming criticism of this feature, I'm > going to shelve it > for now. Damn! That's a real shame! I was really starting to look forward to that. Please reconsider??! > --matt > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discu > ss > Celso This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Proposal: multiple copies of user data
On 12/09/06, Celso <[EMAIL PROTECTED]> wrote: One of the great things about zfs, is that it protects not just against mechanical failure, but against silent data corruption. Having this available to laptop owners seems to me to be important to making zfs even more attractive. I'm not arguing against that. I was just saying that *if* this was useful to you (and you were happy with the dubious resilience/performance benefits) you can already create mirrors/raidz on a single disk by using partitions as building blocks. There's no need to implement the proposal to gain that. Am I correct in assuming that having say 2 copies of your "documents" filesystem means should silent data corruption occur, your data can be reconstructed. So that you can leave your os and base applications with 1 copy, but your important data can be protected. Yes. -- Rasputin :: Jack of All Trades - Master of Nuns http://number9.hellooperator.net/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user data
Take this for what it is: the opinion on someone who knows less about zfs than probably anyone else on this thread ,but... I would like to add my support for this proposal. As I understand it, the reason for using ditto blocks on metadata, is that maintaining their integrity is vital for the health of the filesystem, even if the zpool isn't mirrored or redundant in any way ie laptops, or people who just don't or can't add another drive. One of the great things about zfs, is that it protects not just against mechanical failure, but against silent data corruption. Having this available to laptop owners seems to me to be important to making zfs even more attractive. Granted, if you are running a enterprise based fileserver, this probably isn't going to be your first choice for data protection. You will probably be using the other features of zfs like mirroring, raidz raidz2 etc. Am I correct in assuming that having say 2 copies of your "documents" filesystem means should silent data corruption occur, your data can be reconstructed. So that you can leave your os and base applications with 1 copy, but your important data can be protected. In a way, this reminds me of intel's "matrix raid" but much cooler (it doesn't rely on a specific motherboard for one thing). I would also agree that utilities like 'ls' and quotas should report both and count against peoples quotas. It just doesn't seem to hard to me to understand that because you have 2 copies, you halve the amount of available space. Just to reiterate, I think this would be an awesome feature! Celso. PS. Please feel free to correct me on any technical inaccuracies. I am trying to learn about zfs and Solaris 10 in general. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Proposal: multiple copies of user data
On Tue, 12 Sep 2006, Anton B. Rang wrote: reformatted > >True - I'm a laptop user myself. But as I said, I'd assume the whole disk > >would fail (it does in my experience). Usually a laptop disk suffers a mechanical failure - and the failure rate is a lot higher than disks in a fixed location environment. > That's usually the case, but single-block failures can occur as well. > They're rare (check the "uncorrectable bit error rate" specifications) > but if they happen to hit a critical file, they're painful. > > On the other hand, multiple copies seems (to me) like a really expensive > way to deal with this. ZFS is already using relatively large blocks, so > it could add an erasure code on top of them and have far less storage > overhead. If the assumed problem is multi-block failures in one area of > the disk, I'd wonder how common this failure mode is; in my experience, > multi-block failures are generally due to the head having touched the > platter, in which case the whole drive will shortly fail. (In any case, The following is based on dated knowledge from personal experience and I can't say if its (still) accurate information today. Drive failures in a localized area are generally caused by the heads being positioned in the same (general) cylinder position for long periods of time. The heads ride on a air bearing - but there is still a lot of friction caused by the movement of air under the heads. This is turn generates heat. Localized heat buildup can cause some of the material coated on the disk to break free. The drive is designed for this eventuality - since it is equipped with a very fine filter which will catch and trap anything that breaks free and the airflow is designed to constantly circulate the air through the filter. However, some of the material might get trapped between the head and the disk and possibly stick to the disk. In this case, the neighbouring disk cylinders in this general area will probably be damaged and, if enough material accumulates, so might the head(s). In the old days people wrote their own head "floater" programs - to ensure that the head was moved randomly across the disk surface from time to time. I don't know if this is still relevant today - since the amount of firmware a disk drive executes, continues to increase every day. But in a typical usage scenario, where a user does, for example, a find operation in a home directory - and the directory caches are not sized large enough, there is a good probability that the heads will end up in the same general area of the disk, after the find op completes. Assuming that the box has enough memory, the disk may not be accessed again for a long time - and possibly only during another find op (wash, rinse, repeat). Continuing: a buildup of heat in a localized cylinder area, will cause the disk platter to expand and shift, relative to the heads. The disk platter has one surface dedicated to storing servo information - and from this the disk can "decide" that it is on the wrong cylinder after a head movement. In which case the drive will recalibrate itself (thermal recalibration) and store a table of offsets for different cylinder ranges. So when the head it told, for example, to move to cylinder 1000, the correction table will tell it to move to where physical cylinder 1000 should be and then add the correction delta (plus or minus) for that cylinder range to figure out where to the actually move the heads to. Now the heads are positioned on the correct cylinder and should be centered on it. If the drive gets a bad CRC after reading a cylinder it can use the CRC to correct the data or it can command that the data be re-read, until a correctable read is obtained. Last I heard, the number of retries is of the order of 100 to 200 or more(??). So this will be noticable - since 100 reads will require 100 revolutions of the disk. Retries like this will probably continue to provide correctable data to the user and the disk drive will ignore the fact that there is an area of disk where retries are constantly required. This is what Steve Gibson picked up on for his SpinRite product. If he runs code that can determine that CRC corrections or re-reads are required to retrieve good data, then he "knows" this is a likely area of the disk to fail in the (possibly near) future. So he relocates the data in this area, marks the area "bad", and the drive avoids it. Given what I wrote earlier, that there could be some physical damage in this general area - having the heads avoid it is a Good Thing. So the question is, how relevant is storing multiple copies of data on a disk in terms of the mechanics of modern disk drive failure modes. Without some "SpinRite" like functionality in the code, the drive will continue to access the deteriorating disk cylinders, now a localized failure, and eventually it will deteriorate further and cause enough material to break free to take out the head(s). At which time the
[zfs-discuss] Re: Proposal: multiple copies of user data
>True - I'm a laptop user myself. But as I said, I'd assume the whole disk >would fail (it does in my experience). That's usually the case, but single-block failures can occur as well. They're rare (check the "uncorrectable bit error rate" specifications) but if they happen to hit a critical file, they're painful. On the other hand, multiple copies seems (to me) like a really expensive way to deal with this. ZFS is already using relatively large blocks, so it could add an erasure code on top of them and have far less storage overhead. If the assumed problem is multi-block failures in one area of the disk, I'd wonder how common this failure mode is; in my experience, multi-block failures are generally due to the head having touched the platter, in which case the whole drive will shortly fail. (In any case, multi-block failures could be addressed by spreading the data from a large block and using an erasure code.) This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Re: Proposal: multiple copies of user data
Anton B. Rang wrote: The biggest problem I see with this is one of observability, if not all of the data is encrypted yet what should the encryption property say ? If it says encryption is on then the admin might think the data is "safe", but if it says it is off that isn't the truth either because some of it maybe still encrypted. From a user interface perspective, I'd expect something like Encryption: Being enabled, 75% complete or Encryption: Being disabled, 25% complete, about 2h23m remaining and if we are still writing to the file systems at that time ? Maybe this really does need to be done with the file system locked. I'm not sure how you'd map this into a property (or several), but it seems like "on"/"off" ought to be paired with "transitioning to on"/"transitioning to off" for any changes which aren't instantaneous. Agreed, and checksum and compression would have the same issue if there was a mechanism to rewrite with the new checksums or compression settings. -- Darren J Moffat ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user data
>The biggest problem I see with this is one of observability, if not all >of the data is encrypted yet what should the encryption property say ? >If it says encryption is on then the admin might think the data is >"safe", but if it says it is off that isn't the truth either because >some of it maybe still encrypted. >From a user interface perspective, I'd expect something like Encryption: Being enabled, 75% complete or Encryption: Being disabled, 25% complete, about 2h23m remaining I'm not sure how you'd map this into a property (or several), but it seems like "on"/"off" ought to be paired with "transitioning to on"/"transitioning to off" for any changes which aren't instantaneous. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user data
> Hi Matt, > Interesting proposal. Has there been any > consideration if free space being reported for a ZFS > filesystem would take into account the copies > setting? > > Example: > zfs create mypool/nonredundant_data > zfs create mypool/redundant_data > df -h /mypool/nonredundant_data > /mypool/redundant_data >(shows same amount of free space) > zfs set copies=3 mypool/redundant_data > > Would a new df of /mypool/redundant_data now show a > different amount of free space (presumably 1/3 if > different) than /mypool/nonredundant_data? As I understand the proposal, there's nothing new to do here. The filesystem might be 25% full, and it would be 25% full no matter how many copies of the filesystem there are. Similarly with quotas, I'd argue that the extra copies should not count towards a user's quota, since a quota is set on the filesystem. If I'm using 500M on a filesystem, I only have 500M of data no matter how many copies of it the administrator has decided to keep (cf. RAID1). I also don't see why a copy can't just be dropped if the "copies" value is decreased. Having said this, I don't see any value in the proposal at all, to be honest. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Re: Proposal: multiple copies of user data
Hi Matt, Interesting proposal. Has there been any consideration if free space being reported for a ZFS filesystem would take into account the copies setting? Example: zfs create mypool/nonredundant_data zfs create mypool/redundant_data df -h /mypool/nonredundant_data /mypool/redundant_data (shows same amount of free space) zfs set copies=3 mypool/redundant_data Would a new df of /mypool/redundant_data now show a different amount of free space (presumably 1/3 if different) than /mypool/nonredundant_data? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
