Re: [zfs-discuss] ZFS for write-only media?
"Richard L. Hamilton" <[EMAIL PROTECTED]> wrote: > > I know two write-only device types: > > > > WOM Write-only media > > WORNWrite-once read never (this one is often used > > for backups ;-) > > > > Jörg > > Save $$ (or ??) - use /dev/null instead. See: ftp://ftp.berlios.de/pub/star/testscripts/zwicky/ The Zwicky test claims that that most popular backup program is called "no backup" ;-) Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Thu, 24 Apr 2008, Daniel Rock wrote: > Joerg Schilling schrieb: >> WOM Write-only media > > http://www.national.com/rap/files/datasheet.pdf I love this part of the specification: Cooling The 25120 is easily cooled by employment of a six-foot fan, 1/2" from the package. If the device fails, you have exceeded the ratings. In such cases, more air is recommended. There was an article in German c't magazine's issue exactly 13 years ago this month that benchmarked various operating system's Null devices. They tested an unnamed "hardware null device prototype", now I finally know what that one actually was ! :-) FrankH. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Joerg Schilling schrieb: > WOM Write-only media http://www.national.com/rap/files/datasheet.pdf Daniel ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
> "Dana H. Myers" <[EMAIL PROTECTED]> wrote: > > > Bob Friesenhahn wrote: > > > Are there any plans to support ZFS for write-only > media such as > > > optical storage? It seems that if mirroring or > even zraid is used > > > that ZFS would be a good basis for long term > archival storage. > > I'm just going to assume that "write-only" here > means "write-once, > > read-many", since it's far too late for an April > Fool's joke. > > I know two write-only device types: > > WOM Write-only media > WORN Write-once read never (this one is often used > for backups ;-) > > Jörg Save $$ (or €€) - use /dev/null instead. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Bob Friesenhahn <[EMAIL PROTECTED]> wrote: > On Mon, 21 Apr 2008, Dana H. Myers wrote: > > > Bob Friesenhahn wrote: > >> Are there any plans to support ZFS for write-only media such as optical > >> storage? It seems that if mirroring or even zraid is used that ZFS would > >> be a good basis for long term archival storage. > > I'm just going to assume that "write-only" here means "write-once, > > read-many", since it's far too late for an April Fool's joke. > > Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies > such as holographic drives (300GB drives are on the market). For > example, with two CD-R drives it should be possible to build a ZFS > mirror on two CDs, but the I/O to these devices may need to be done in > a linear sequential fashion at a rate sufficient to keep the writer > happy, so temporary files (or memory-based buffering) likely need to > be used. CD-R media is not really WORM (write once read many) as CD-R does not allow to write _every_ sector exactly once. Due to the way scrambeled error correction is implemented, there is always unusable 7 sectors between two areas on the medium that have been written independently. Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
"Dana H. Myers" <[EMAIL PROTECTED]> wrote: > Bob Friesenhahn wrote: > > Are there any plans to support ZFS for write-only media such as > > optical storage? It seems that if mirroring or even zraid is used > > that ZFS would be a good basis for long term archival storage. > I'm just going to assume that "write-only" here means "write-once, > read-many", since it's far too late for an April Fool's joke. I know two write-only device types: WOM Write-only media WORNWrite-once read never (this one is often used for backups ;-) Jörg -- EMail:[EMAIL PROTECTED] (home) Jörg Schilling D-13353 Berlin [EMAIL PROTECTED](uni) [EMAIL PROTECTED] (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/old/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Tue, 22 Apr 2008, Jonathan Loran wrote: >> > I suppose with ditto blocks, this has some merrit. Someone needs to > characterize how errors probigate on different types of WORM media. perhaps > this has already been done. In my experience, when DVD-R go south, they > really go bad at once. Not a lot of small bit errors. But a full analysis > would be good. Probably it would make the most sence to write mirrored WORM > disks with different technology to hedge your bets. It does not really matter since ZFS supports various forms of RAID, including arbitrary mirroring. If possible, the media can be purchased from different vendors so there is less chance of similar bit-rot across the lot. With $40 to $200 million spent per project, a few extra copies is in the noise. :-) Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
2008-04-22
Thread
Peter Brouwer, Principal Storage Architect, Office of the Chief Technologist, Sun MicroSystems
Ralf Bertling wrote: I am not an expert, but the MTTDL is in tousands of years when using raidz2 with a hot-spare and regular scrubbing. If you add zpool send/receive and geographically dislocated severs, this may be better than optical media, because you detect the errors early. Two things to be aware of with this type of statistical info ( also true for MTBF ), It is statistical info and it is an average, it does not give any guarantee that a catastrophic event will not happen in the next hr. Secondly this figure is defined on subsystem level, if the box goes up in smoke the thousands of years MTTDL is of no use! So the moral of the story is to always plan for a second copy of your data if you want to design resilience from a data set point of view. -- Regards Peter Brouwer, Sun Microsystems Linlithgow Principal Storage Architect, ABCP DRII Consultant Office:+44 (0) 1506 672767 Mobile:+44 (0) 7720 598226 Skype :flyingdutchman_,flyingdutchman_l smime.p7s Description: S/MIME Cryptographic Signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Bob Friesenhahn wrote: > On Tue, 22 Apr 2008, Jonathan Loran wrote: >>> >> But that's the point. You can't correct silent errors on write once >> media because you can't write the repair. > > Yes, you can correct the error (at time of read) due to having both > redundant media, and redundant blocks. That is a normal function of > ZFS. It just not possible to correct the failed block on the media by > re-writing it or moving its data to a new location. > I suppose with ditto blocks, this has some merrit. Someone needs to characterize how errors probigate on different types of WORM media. perhaps this has already been done. In my experience, when DVD-R go south, they really go bad at once. Not a lot of small bit errors. But a full analysis would be good. Probably it would make the most sence to write mirrored WORM disks with different technology to hedge your bets. Jon ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Tue, 22 Apr 2008, Jonathan Loran wrote: >> > But that's the point. You can't correct silent errors on write once > media because you can't write the repair. Yes, you can correct the error (at time of read) due to having both redundant media, and redundant blocks. That is a normal function of ZFS. It just not possible to correct the failed block on the media by re-writing it or moving its data to a new location. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Bob Friesenhahn wrote: >> The "problem" here is that by putting the data away from your machine, >> you loose the chance to "scrub" >> it on a regular basis, i.e. there is always the risk of silent >> corruption. >> > > Running a scrub is pointless since the media is not writeable. :-) > > But that's the point. You can't correct silent errors on write once media because you can't write the repair. I think it makes more sense to save a checksum of the entire CD/DVD/etc. media separately, so you can check the validity of your data that way, instead of using ZFS on WORM media. Jon ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Tue, 22 Apr 2008, Ralf Bertling wrote: > Hi Bob, > If I was willing to do that I would simply build a pool from file- > based storage being n-ISO images. > It would involve the following steps > 1. create blank ISO images of the size of your media > 2. zpool create wormyz raidz2 image1.iso image2.iso image3.iso ... > 3. Move your data to the pool > 4. export the pool > 5. burn the media > > If you need to recover, copy the data from the device using dd > conv=sync,noerror Yes, I know that this will work and what I thought of. But I was thinking that perhaps ZFS would be able to attach to the read-only pool. At the moment it is likely not willing to attach to read-only devices since part of its function depends on writing. > The "problem" here is that by putting the data away from your machine, > you loose the chance to "scrub" > it on a regular basis, i.e. there is always the risk of silent > corruption. Running a scrub is pointless since the media is not writeable. :-) > I am not an expert, but the MTTDL is in tousands of years when using > raidz2 with a hot-spare and regular scrubbing. A thousand years ago, knights were storming castle calls. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Hi Bob, If I was willing to do that I would simply build a pool from file- based storage being n-ISO images. It would involve the following steps 1. create blank ISO images of the size of your media 2. zpool create wormyz raidz2 image1.iso image2.iso image3.iso ... 3. Move your data to the pool 4. export the pool 5. burn the media If you need to recover, copy the data from the device using dd conv=sync,noerror The "problem" here is that by putting the data away from your machine, you loose the chance to "scrub" it on a regular basis, i.e. there is always the risk of silent corruption. I am not an expert, but the MTTDL is in tousands of years when using raidz2 with a hot-spare and regular scrubbing. If you add zpool send/receive and geographically dislocated severs, this may be better than optical media, because you detect the errors early. Hope this helps, ralf Am 21.04.2008 um 23:18 schrieb [EMAIL PROTECTED]: > Date: Mon, 21 Apr 2008 14:20:15 -0500 (CDT) > From: Bob Friesenhahn <[EMAIL PROTECTED]> > Subject: [zfs-discuss] ZFS for write-only media? > To: zfs-discuss@opensolaris.org > Message-ID: > <[EMAIL PROTECTED]> > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed > > Are there any plans to support ZFS for write-only media such as > optical storage? It seems that if mirroring or even zraid is used > that ZFS would be a good basis for long term archival storage. > > Has this been considered? I expect that it is possible today by using > files as the underlying media and then copying those individual files > to optical storage. > > Bob > == > Bob Friesenhahn > [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
> Are there any plans to support ZFS for write-only > media such as > optical storage? It seems that if mirroring or even > zraid is used > that ZFS would be a good basis for long term archival > storage. A simple solution would be to set up a zpool using two devices (real or ZVOL) that are the size of the optical media that you will be using, offline it when you're done copying files to it, and then write the images to optical media. To restore later on, just reverse the process. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Bob, ADM, with an underlying ZFS filesystem, will provide a great archiving solution when fully implemented. It allows for file replication, migration from disk to tape/optical/more disk, copies any number you may want, and allows for policy management of the archives. It's underlying media management is provided by MMS (http://www.opensolaris.org/os/project/mms/). We follow the standards are are open sourced. - Greg Matthews, Archive Products Group, Sun Microsystems, Inc. Bob Friesenhahn wrote: > On Mon, 21 Apr 2008, Mark A. Carlson wrote: > > >> Interesting problem. And yes you are right, there are a number >> of problems to solve here, see: >> >> http://blogs.sun.com/mac/en_US/entry/open_archive >> > > Standards and open source are clearly the way to go. Many open source > applications have already been demonstrated to last far longer than > their commercial counterparts. > > ZFS is open sourced but it is perhaps not mature and widespread enough > yet to be seen as a stable long-term storage standard. The problem is > a long term problem so there seems to be opportunity here for ZFS if > it is adapted somewhat to address archiving. > > Bob > == > Bob Friesenhahn > [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ > > ___ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Mon, 21 Apr 2008, Mark A. Carlson wrote: > Interesting problem. And yes you are right, there are a number > of problems to solve here, see: > > http://blogs.sun.com/mac/en_US/entry/open_archive Standards and open source are clearly the way to go. Many open source applications have already been demonstrated to last far longer than their commercial counterparts. ZFS is open sourced but it is perhaps not mature and widespread enough yet to be seen as a stable long-term storage standard. The problem is a long term problem so there seems to be opportunity here for ZFS if it is adapted somewhat to address archiving. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Interesting problem. And yes you are right, there are a number of problems to solve here, see: http://blogs.sun.com/mac/en_US/entry/open_archive -- mark Bob Friesenhahn wrote: On Mon, 21 Apr 2008, Mark A. Carlson wrote: Maybe what you want is to archive files off to optical media? Perhaps ADM - http://opensolaris.org/os/project/adm ? That looks interesting, but true archiving is needed. The level of archiving for this application is that copies would be kept thousands of feet underground in a stable salt mine on continents 'A' and 'B'. An alternative is special temperature, humidity, and pressure controlled above-ground bunkers. It is desired that the data be preserved for hundreds or a thousand years, which would of course require copying to more modern media ever so often. The cost to create the original data is up to $200 million (today's cost) and it can not be recreated. The size of the originals to be archived ranges from 2TB to 400TB depending on how "deep" the archiving is. The existing archive approach is in analog form but it is found that there is noticeable degredation after 50 or 100 years which is not possible to fully correct. When saw a discussion of these requirements today, ZFS immediately came to mind due to its many media-independent error detection and correction features, and the fact that it is open source. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Mon, 21 Apr 2008, Mark A. Carlson wrote: > Maybe what you want is to archive files off to optical media? > > Perhaps ADM - http://opensolaris.org/os/project/adm ? That looks interesting, but true archiving is needed. The level of archiving for this application is that copies would be kept thousands of feet underground in a stable salt mine on continents 'A' and 'B'. An alternative is special temperature, humidity, and pressure controlled above-ground bunkers. It is desired that the data be preserved for hundreds or a thousand years, which would of course require copying to more modern media ever so often. The cost to create the original data is up to $200 million (today's cost) and it can not be recreated. The size of the originals to be archived ranges from 2TB to 400TB depending on how "deep" the archiving is. The existing archive approach is in analog form but it is found that there is noticeable degredation after 50 or 100 years which is not possible to fully correct. When saw a discussion of these requirements today, ZFS immediately came to mind due to its many media-independent error detection and correction features, and the fact that it is open source. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Maybe what you want is to archive files off to optical media? Perhaps ADM - http://opensolaris.org/os/project/adm ? -- mark Bob Friesenhahn wrote: On Mon, 21 Apr 2008, Dana H. Myers wrote: Bob Friesenhahn wrote: Are there any plans to support ZFS for write-only media such as optical storage? It seems that if mirroring or even zraid is used that ZFS would be a good basis for long term archival storage. I'm just going to assume that "write-only" here means "write-once, read-many", since it's far too late for an April Fool's joke. Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies such as holographic drives (300GB drives are on the market). For example, with two CD-R drives it should be possible to build a ZFS mirror on two CDs, but the I/O to these devices may need to be done in a linear sequential fashion at a rate sufficient to keep the writer happy, so temporary files (or memory-based buffering) likely need to be used. No one wants to be faced with a situation in which two copies are made to CD but both copies are deemed to be bad when they are read. ZFS could make that situation much better. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
On Mon, 21 Apr 2008, Dana H. Myers wrote: > Bob Friesenhahn wrote: >> Are there any plans to support ZFS for write-only media such as optical >> storage? It seems that if mirroring or even zraid is used that ZFS would >> be a good basis for long term archival storage. > I'm just going to assume that "write-only" here means "write-once, > read-many", since it's far too late for an April Fool's joke. Yes, of course. Such as to CD-R, DVD-RW, or more exotic technologies such as holographic drives (300GB drives are on the market). For example, with two CD-R drives it should be possible to build a ZFS mirror on two CDs, but the I/O to these devices may need to be done in a linear sequential fashion at a rate sufficient to keep the writer happy, so temporary files (or memory-based buffering) likely need to be used. No one wants to be faced with a situation in which two copies are made to CD but both copies are deemed to be bad when they are read. ZFS could make that situation much better. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS for write-only media?
Bob Friesenhahn wrote: > Are there any plans to support ZFS for write-only media such as > optical storage? It seems that if mirroring or even zraid is used > that ZFS would be a good basis for long term archival storage. I'm just going to assume that "write-only" here means "write-once, read-many", since it's far too late for an April Fool's joke. ;-) Dana ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS for write-only media?
Are there any plans to support ZFS for write-only media such as optical storage? It seems that if mirroring or even zraid is used that ZFS would be a good basis for long term archival storage. Has this been considered? I expect that it is possible today by using files as the underlying media and then copying those individual files to optical storage. Bob == Bob Friesenhahn [EMAIL PROTECTED], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
