Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 21/01/2010 11:55, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings up full circle... What do you then use to backup to tape bearing in mind that the Sun-provided tools all have significant limitations? I guess you need to use a third party tool and watch carefully that they provide complete backups. in this specific environment I was talking about NetBackup is used. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
uep, This solution seems like the best and most efficient way of handling large filesystems. My biggest question however is, when backing this up to tape, can it be split across several tapes? I will be using bacula to back this up. Will i need to tar or star this filesystem before writing it to tape? The next question I have is since I am using a primary and a secondary server, using zfs send/recv how would I incorporate this solution to then backing the secondary SAN to tape? Will I need double the hard disk space as I will need a file based copy of the ZFS filesystem? I know it is a lot of questions but I thought the solution would work perfect in my environment. Thanks, Greg -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Thu, Jan 21, 2010 at 11:28 AM, Richard Elling richard.ell...@gmail.com wrote: On Jan 21, 2010, at 3:55 AM, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings up full circle... What do you then use to backup to tape bearing in mind that the Sun-provided tools all have significant limitations? Poor choice of words. Sun resells NetBackup and (IIRC) that which was formerly called NetWorker. Thus, Sun does provide enterprise backup solutions. (Symantec nee Veritas) NetBackup and (EMC nee Legato) Networker are different products that compete in the enterprise backup space. Under the covers NetBackup uses gnu tar to gather file data for the backup stream. At one point (maybe still the case), one of the claimed features of netbackup is that if a tape is written without multiplexing, you can use gnu tar to extract data. This seems to be most useful when you need to recover master and/or media servers and to be able to extract your data after you no longer use netbackup. -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins wrote: True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. What kind of removable hard drives are you getting that are cheaper than tape? LTO4 media is less than 2.5 cents/GB for us (before compression, acquisition cost only). -- Darren ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Thu, Jan 21, 2010 at 12:38:56AM +0100, Ragnar Sundblad wrote: On 21 jan 2010, at 00.20, Al Hopper wrote: I remember for about 5 years ago (before LT0-4 days) that streaming tape drives would go to great lengths to ensure that the drive kept streaming - because it took so much time to stop, backup and stream again. And one way the drive firmware accomplished that was to write blocks of zeros when there was no data available. I haven't seen drives that fill out with zeros. Sounds like an ugly solution, but maybe it could be useful in some strange case. It was closer to 15 years ago than 5, but this may be a reference to the first release of the DLT 7000. That version came out with only 4MB as a RAM buffer, which is insufficient to buffer at speed during a stop/start cycle. It didn't write zeros, but it would disable the on-drive compression to try to keep the speed of bits being written to tape up. So the effect was similar in that the capacity of the media was reduced. The later versions had 8MB buffers and that behavior was removed. -- Darren ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
A Darren Dunham wrote: On Wed, Jan 20, 2010 at 08:11:27AM +1300, Ian Collins wrote: True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. What kind of removable hard drives are you getting that are cheaper than tape? It's not the raw cost per GB, it's the way the tapes are used. To aid recovery times, a number of different backup sets (groups of filesystems) are written, so the tapes aren't all used to capacity. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20/01/2010 15:45, David Dyer-Bennet wrote: On Wed, January 20, 2010 09:23, Robert Milkowski wrote: Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. Is there an rsync out there that can reliably replicate all file characteristics between two ZFS/Solaris systems? I haven't found one. The ZFS ACLs seem to be beyond all of them, in particular. No it doesn't support ZFS ACLs - fortunately it is not an issue for us. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. A more apples to apples comparison would be to compare the storage only. Both removable drive and tape options require a server with FC or SCSI ports, so that can be excluded from the comparison. I think one should actually compare whole solutions - including servers, fc infrastructure, tape drives, robots, software costs, rack space, ... Servers like x4540 are ideal for zfs+rsync backup solution - very compact, good $/GB ratio, enough CPU power for its capacity, allow to easily scale it horizontally, and it is not too small and not too big. Then thanks to its compactness they are very easy to administer. Depending on an anvironment one could deploy them always in paris - one in one datacenter and 2nd one in other datacanter with ZFS send based replication of all backups (snapshots). Or one may replicate (cross-replicate) only selected clients if needed. -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Robert Milkowski wrote: On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. A more apples to apples comparison would be to compare the storage only. Both removable drive and tape options require a server with FC or SCSI ports, so that can be excluded from the comparison. I think one should actually compare whole solutions - including servers, fc infrastructure, tape drives, robots, software costs, rack space, ... Servers like x4540 are ideal for zfs+rsync backup solution - very compact, good $/GB ratio, enough CPU power for its capacity, allow to easily scale it horizontally, and it is not too small and not too big. Then thanks to its compactness they are very easy to administer. Until you try to pick one up and put it in a fire safe! Depending on an anvironment one could deploy them always in paris - one in one datacenter and 2nd one in other datacanter with ZFS send based replication of all backups (snapshots). Or one may replicate (cross-replicate) only selected clients if needed. Yes, I agree. That's how my client's systems are configured (pairs). We also have another with an attached tape library. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Robert Milkowski wrote: I think one should actually compare whole solutions - including servers, fc infrastructure, tape drives, robots, software costs, rack space, ... Servers like x4540 are ideal for zfs+rsync backup solution - very compact, good $/GB ratio, enough CPU power for its capacity, allow to easily scale it horizontally, and it is not too small and not too big. Then thanks to its compactness they are very easy to administer. Depending on an anvironment one could deploy them always in paris - one in one datacenter and 2nd one in other datacanter with ZFS send based replication of all backups (snapshots). Or one may replicate (cross-replicate) only selected clients if needed. Something else that often sells the 4500/4540 relates to internal company politics. Often, inside a company storage has to be provisioned from the company's storage group, using very expensive SAN based storage, indeed so expensive by the time the company's storage group have added their overhead onto the already expensive SAN, that whole projects become unviable. Instead, teams find they can order 4500/4540's which slip under the radar as servers (or even PCs), and they now have affordable storage for their projects, which makes them viable once more. -- Andrew ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 21/01/2010 09:07, Ian Collins wrote: Robert Milkowski wrote: On 20/01/2010 19:20, Ian Collins wrote: Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. A more apples to apples comparison would be to compare the storage only. Both removable drive and tape options require a server with FC or SCSI ports, so that can be excluded from the comparison. I think one should actually compare whole solutions - including servers, fc infrastructure, tape drives, robots, software costs, rack space, ... Servers like x4540 are ideal for zfs+rsync backup solution - very compact, good $/GB ratio, enough CPU power for its capacity, allow to easily scale it horizontally, and it is not too small and not too big. Then thanks to its compactness they are very easy to administer. Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. :) ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings up full circle... What do you then use to backup to tape bearing in mind that the Sun-provided tools all have significant limitations? I guess you need to use a third party tool and watch carefully that they provide complete backups. JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 21, 2010, at 3:55 AM, Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings up full circle... What do you then use to backup to tape bearing in mind that the Sun-provided tools all have significant limitations? Poor choice of words. Sun resells NetBackup and (IIRC) that which was formerly called NetWorker. Thus, Sun does provide enterprise backup solutions. If I may put on my MBA hat, the competition is not ufsdump. ufsdump has nearly zero market penetration and no prospects for improving its market share. Making another ufsdump will also gain no market share. The market leaders are the likes of EMC, IBM, and Symantec with their heterogenous backup support. If Sun wanted to provide a better solution that might gain market share against the others, then it would also need to be heterogenous. So I think it would be hard to make a business case for a whole new backup solution. A less costly and less risky approach is to work with the market leaders to better integrate with dataset replication. Caveat: this may already be available, I haven't looked recently. I guess you need to use a third party tool and watch carefully that they provide complete backups. This is a good idea anyway. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel wrote: Until you try to pick one up and put it in a fire safe! Then you backup to tape from x4540 whatever data you need. In case of enterprise products you save on licensing here as you need a one client license per x4540 but in fact can backup data from many clients which are there. Which brings up full circle... What do you then use to backup to tape bearing in mind that the Sun-provided tools all have significant limitations? In addition to Richard's comments, I doubt many medium to large businesses would use ufsdump/restore as their backup solution. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Allen Eastwood wrote: On Jan 19, 2010, at 22:54 , Ian Collins wrote: Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup solutions on snapshots, I assert the problem is low priority. What I have issue with is the idea that no one uses/should use tape any more. There are places for tape and it still has value as a backup device. In many cases in the past, ufsdump, despite it's many issues, was able to restore working OS's, or individual files. Perfect, not by a long shot. But it did get the job done. As was pointed out earlier, all I needed was a Solaris CD (or network boot) and I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted, same thing…and I can restore just the file(s) I need. And while it's been a few years since I've read the man page on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing with a downed system. For a full recovery, you can archive a send stream and receive it back. With ZFS snapshots, the need for individual file recovery from tape is much reduced. The backup server I manage for a large client has 60 days of snaps and I can't remember when they had to go to tape to recover a file. -- Ian. Let's see… For full recovery, I have to zfs send to something, preferably that understands tape (yes, I know I can send to tape directly, but how well does zfs send handle the end of the tape? auto-changers?). I keep a stream (as a file) of my root pool on a USB stick. It could be on tape, but root pools are small. Then for individual file recovery, I have snaphots…which I also have to get on to tape…if I want to have them available on something other than the boot devices. No, just keep the snapshots in place. If a file is lost, just gab it form the snapshot directory. If the root filesystem is munted, roll back to the last snapshot. Now…to recover the entire OS, perhaps not so bad…but that's one tool. And to recover the one file, say a messed up /etc/system, that's preventing my OS from booting? Have to get that snapshot where I can use it first…oh and restoring individual files and not the entire snapshot? As I said, roll back. Boot form install media, import the root pool, get the file from a snapshot, or roll back to the last good snapshot, export and reboot. At best, it's an unwieldy process. But does it offer the simplicity that ufsdump/ufsrestore (or dump/restore on how many Unix variants…) did? No way. It certainly does for file recovery. Do you run incremental dumps every hour, or every 15 minutes? Periodic snapshots are quick and cheep. As I said before, careful use of snapshots all be removes the need to recover files from tape. We have 60 days of 4 hourly and 24 hourly snapshots in place, so the odds on finding a recent copy of a lost file are way better than they would be on daily incrementals. I certainly don't miss the pain of loading a sequence of incrementals to recover lost data. So ZFS solves the problems in a different way. For fat-finger recovery, it's way better than ufsdump/ufsrestore. A simple, effective dump/restore that deals with all the supported file systems, can deal with tape or disk, and allows for complete OS restore or individual file restore, and can be run from a install CD/DVD. As much as I love ZFS and as many problems as it does solve, leaving this out was a mistake, IMO. It possibly was, but it has encouraged us to find better solutions. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 19 jan 2010, at 20.11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. In the past few years growth in hard drive capacities has outstripped tapes to the extent that removable hard drives and ZFS snapshots have become a more cost effective and convenient backup media. LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. LTO drivers cost a little, but so do disk shelves. So, now that there is no big price issue, there is choice instead. Use it! Hard drives are good for random access - both restore of individual files and partial rewrite. Hard drivers aren't faster than tape for data transfer, but they might be cheaper to run in parallel and therefore you could potentially gain speed. Hard drives have shorter seek time, which may be important. Hard drives are probably bad for storing for longer times - especially - you will never know how long it could be stored before it will fail. A month? Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes are supposed to be able to keep it's data for at least 30 years of stored properly. Hard drives are probably best when used online or at least very often. So - it is wrong to say that one is better or cheaper than the other. They have different properties, and could be used to solve different problems. /ragge s ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Richard Elling richard.ell...@gmail.com wrote: ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. How quickly we forget ufsdump's limitations :-). For example, it is not supported for use on an active file system (known data corruption possibility) and UFS snapshots are, well, a poor hack and often not usable for backups. As the ufsdump(1m) manpage says, It seems you forgot that zfs also needs snapshots. There is nothing bad with snapshots. When I was talking with Jeff Bonwick in September 2004 (before ZFS became public), the only feature that was missing in Solaris for a 100% correct backup based on star was an interface for holey files, so we designed it. I believe the only mistake from ufsdumps is that is does not use standard OS interfaces and that it does not use a standard archive format. You get both with star and star is even faster than ufsdump. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Ian Collins i...@ianshome.com wrote: The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g. ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz The ACL format used by Sun is undocumented.. man acltotext We are talking about TAR and I did give a pointer to the star archive format documentation, so it is obvious that I was talking about the ACL format from Sun tar. This format is not documented. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Edward Ned Harvey sola...@nedharvey.com wrote: Star implements this in a very effective way (by using libfind) that is even faster that the find(1) implementation from Sun. Even if I just find my filesystem, it will run for 7 hours. But zfs can create my whole incremental snapshot in a minute or two. There is no way star or any other user-space utility that walks the filesystem can come remotely close to this performance. Such performance can only be implemented at the filesystem level, or lower. You claim that it is fast for you but this is because it is block oriented and because you probably changed only few data. If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star. Note that ufsdump directly accesees the raw disk device and thus _is_ at filesystem leven but still is slower than star on UFS. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
While I can appreciate that ZFS snapshots are very useful in being able to recover files that users might have deleted, they do not do much to help when the entire disk array experiences a crash/corruption or catches fire. Backing up to a second array helps if a) the array is off-site and for many of us the cost of remote links with sufficient bandwidth is still prohibitive,or b) on the local network but sufficiently far away from the original array such that the fire does not corrupt damage the backup as well. This leaves some form of removable storage. I'm not sure I'm aware of any enterprise-level removable disk solution, primarily because disk isn't really designed to be used for offsite backup whereas tape is. The biggest problem with tape was finding a sufficiently large window in which to perform the backup. ZFS snapshots completely solves this issue, but Sun have failed to provide the mechanism to protect the data off-site. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star. Note that ufsdump directly accesees the raw disk device and thus _is_ at filesystem leven but still is slower than star on UFS. While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. If our customers find a bug in their backup that is caused by a failure in a Sun supplied utility, then they have a legal course of action. The customer's system administrators are covered because they were using tools provided by the vendor. The wrath of the customer would be upon Sun, not the supplier (us) or the supplier's technical lead (me). If the system administrator has chosen star (or if the supplier recommends star), then the conversation becomes a lot more awkward. From the perspective of the business, the system administrator will have acted irresponsibly by choosing a tool that has no vendor support. Alternatively, the supplier will be held responsible for recommending a product that has broken the customer's ability to restore, and with no legal recourse, I wouldn't dare touch it. Sorry. This is why Sun need to provide the solution themselves (or adopt and provide support for star or similar third party products). JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star. Note that ufsdump directly accesees the raw disk device and thus _is_ at filesystem leven but still is slower than star on UFS. While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. If our customers find a bug in their backup that is caused by a failure in a Sun supplied utility, then they have a legal course of action. The customer's system administrators are covered because they were using tools provided by the vendor. The wrath of the customer would be upon Sun, not the supplier (us) or the supplier's technical lead (me). Do you really believe that Sun will help such a customer? There are many bugs in Solaris (I remember e.g. some showstopper bugs in the multimedia area) that are not fixed although they are known since a very long time (more than a year). There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even before) that is not fixed. As a result in many cases ACLs are not restored. Note that bugs in star are fixed much faster and looking back at the 28 years of history with star, I know of not a single bug that took more than 3 months to get a fix. Typically, bugs are fixed withing less than a week - many bugs even within a few hours. This is a support quality that Sun does not offer. So please explain us where you see a problem with star.. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. From the perspective of the business, the contract with Sun provides that support. If our customers find a bug in their backup that is caused by a failure in a Sun supplied utility, then they have a legal course of action. The customer's system administrators are covered because they were using tools provided by the vendor. The wrath of the customer would be upon Sun, not the supplier (us) or the supplier's technical lead (me). Do you really believe that Sun will help such a customer? There are many bugs in Solaris (I remember e.g. some showstopper bugs in the multimedia area) that are not fixed although they are known since a very long time (more than a year). There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even before) that is not fixed. As a result in many cases ACLs are not restored. If Sun don't fix a critical bug that is affecting the availability of server that is under support, then it becomes a problem for the legal department. In the ACL example, it's possible the effected users didn't have a support contract. Note that bugs in star are fixed much faster and looking back at the 28 years of history with star, I know of not a single bug that took more than 3 months to get a fix. Typically, bugs are fixed withing less than a week - many bugs even within a few hours. This is a support quality that Sun does not offer. Possibly, but there is no guarantee that it will be fixed, no-one to call when there is a problem, no-one to escalate the problem to if it is ignored, and no company to sue if it all goes wrong. So please explain us where you see a problem with star.. Hopefully my above comments explain sufficiently. It's not a technical issue with star, it's a business issue. The rules there are very different and not based on merit (this is also why many companies prefer running their mission critical apps on Red Hat Enterprise Linux instead of CentOS, even though technically they are almost identical). JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. From the perspective of the business, the contract with Sun provides that support. From a perspective of reality, such a contract will not help. Do you really believe that Sun will help such a customer? There are many bugs in Solaris (I remember e.g. some showstopper bugs in the multimedia area) that are not fixed although they are known since a very long time (more than a year). There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even before) that is not fixed. As a result in many cases ACLs are not restored. If Sun don't fix a critical bug that is affecting the availability of server that is under support, then it becomes a problem for the legal department. In the ACL example, it's possible the effected users didn't have a support contract. What you seem to point out is that in case of a problem for a customer with a contract, the legal department gets involved. Unfortunately, laywers do not fix bugs. Note that bugs in star are fixed much faster and looking back at the 28 years of history with star, I know of not a single bug that took more than 3 months to get a fix. Typically, bugs are fixed withing less than a week - many bugs even within a few hours. This is a support quality that Sun does not offer. Possibly, but there is no guarantee that it will be fixed, no-one to call when there is a problem, no-one to escalate the problem to if it is ignored, and no company to sue if it all goes wrong. Escalating a problem does not fix it. So please explain us where you see a problem with star.. Hopefully my above comments explain sufficiently. It's not a technical issue with star, it's a business issue. The rules there are very different and not based on merit (this is also why many companies prefer running their mission critical apps on Red Hat Enterprise Linux instead of CentOS, even though technically they are almost identical). Now we are back to reality. A person that is interested in a solution will usually check what happened in similar cases before. If you compare star with Sun supplied tools with this background, Sun cannot outperform star. Red Hat Enterprise Linux may offer something you cannot get with CentOS. But I don't see that Sun can offer something you don't get with star. Let me make another reality check: Many people use GNU tar for backup purposes, but my first automated test case with incremental backups using GNU tar did fail so miserably that I was unable to use GNU tar as a test reference at all. On the other side, I am doing incremental backup _and_ restore tests with gigabytes of real delta data on a dayly base since 2004 and I did hot see any problem since April 2005. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Joerg Schilling wrote: Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: If you like to have a backup that allows to access files, you need a file based backup and I am sure that even a filesystem level scan for recently changed files will not be much faster than what you may achive with e.g. star. Note that ufsdump directly accesees the raw disk device and thus _is_ at filesystem leven but still is slower than star on UFS. While I am sure that star is technically a fine utility, the problem is that it is effectively an unsupported product. From this viewpoint, you may call most of Solaris unsupported. what is that supposed to mean? Michael -- Michael Schusterhttp://blogs.sun.com/recursion Recursion, n.: see 'Recursion' ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 19/01/2010 19:11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. In the past few years growth in hard drive capacities has outstripped tapes to the extent that removable hard drives and ZFS snapshots have become a more cost effective and convenient backup media. What do people with many tens of TB use for backup these days? http://milek.blogspot.com/2009/12/my-presentation-at-losug.html -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20/01/2010 10:48, Ragnar Sundblad wrote: On 19 jan 2010, at 20.11, Ian Collins wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. In the past few years growth in hard drive capacities has outstripped tapes to the extent that removable hard drives and ZFS snapshots have become a more cost effective and convenient backup media. LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. LTO drivers cost a little, but so do disk shelves. So, now that there is no big price issue, there is choice instead. Use it! Hard drives are good for random access - both restore of individual files and partial rewrite. Hard drivers aren't faster than tape for data transfer, but they might be cheaper to run in parallel and therefore you could potentially gain speed. Hard drives have shorter seek time, which may be important. Hard drives are probably bad for storing for longer times - especially - you will never know how long it could be stored before it will fail. A month? Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes are supposed to be able to keep it's data for at least 30 years of stored properly. Hard drives are probably best when used online or at least very often. So - it is wrong to say that one is better or cheaper than the other. They have different properties, and could be used to solve different problems. It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, January 20, 2010 09:23, Robert Milkowski wrote: Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. Is there an rsync out there that can reliably replicate all file characteristics between two ZFS/Solaris systems? I haven't found one. The ZFS ACLs seem to be beyond all of them, in particular. (Losing just that, and preserving the data, is clearly far, far better than losing everything! And a system build *knowing* it was losing the protections could preserve them some other way.) -- David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, January 20, 2010 04:48, Ragnar Sundblad wrote: LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. LTO drivers cost a little, but so do disk shelves. So, now that there is no big price issue, there is choice instead. Use it! Depends on the scale you're operating at. Backing up my 800GB home data pool onto a couple of external 1TB USB drives is *immensely* cheaper than buying tape equipment. At enough bigger scales, I accept that tape is still cheaper. Makes sense, since the tapes are relatively simple compared to drives, and you only need a small number of drives to use a large number of tapes. I think hard drives are still cheaper at small-enterprise levels, actually. -- David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/ Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/ Photos: http://dd-b.net/photography/gallery/ Dragaera: http://dragaera.info ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, 20 Jan 2010, Julian Regel wrote: If our customers find a bug in their backup that is caused by a failure in a Sun supplied utility, then they have a legal course of action. The customer's system administrators are covered because they were using tools provided by the vendor. The wrath of the customer would be upon Sun, not the supplier (us) or the supplier's technical lead (me). I would love to try whatever you are smoking because it must be really good stuff. It would be a bold new step for me, but the benefits are clear. While your notions of the transitive protection offered by vendor support are interesting, I will be glad to meet you in the unemployment line then we can share some coffee and discuss the good old days. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. Which isn't to say tape would be a better solution since it's going to be slower to restore etc. But it does show that tape can work out cheaper, especially since the cost of a high speed WAN link isn't required. JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
ae == Allen Eastwood mi...@paconet.us writes: ic == Ian Collins i...@ianshome.com writes: If people are really still backing up to tapes or DVD's, just use file vdev's, export the pool, and then copy the unmounted vdev onto the tape or DVD. ae And some of those enterprises require backup mechanism that ae can be easily used in a DR situation. ae ufsdump/restore was perfect in that regard. The lack of ae equivalent functionality is a big problem for the situations ae where this functionality is a business requirement. ae For example, one customer, local government, requires a backup ae that can be taken offsite and used in a DR situation. Were you confused by some part of: Use file vdevs, epxort the pool, and then copy the unmounted vdev onto the tape. or do you find that this doesn't do what you want? because it seems fine to me. And the fact that it doesn't need any extra tools means it's unlikely to break (1) far into the future or (2) for a few unlucky builds, and (3) that the restore environment is simple and doesn't involve prepopulating some Legato Database with the TOC of every tape in the library or some such nonsense, which ought to all be among your ``requirements'' but if you're substituting for those, ``works in exactly the way we were used to it working before'' then you may as well use 'zfs send' since you're more concerned with identical-feeling invocatoin syntax than the problems I mentioned. ic For a full recovery, you can archive a send stream and receive ic it back. You can send the stream to the tape, transport the tape to the DR site, and receive it. You can do this weekly as part of your offsite backup plan provided that you receive each tape you transport immediately. Then the data should be permanently stored on disk at the DR site, and the tapes used only for transport. If you store the backup permanently on tape then it's a step backwards from tar/cpio/ufsrestore because the 'zfs send' format is more fragile and has to be restored entire. If you receive the tape immediately this is an improvement because under the old convention tapes could be damaged in transit, or over the years by heat/dust/sunlight, without your knowledge, while on disks it's simple to scrub periodically. I am not trying to take away your tapes, Allen, so please quote Ian instead if that's the thing you object to. I've instead suggested a different way to use them if you really do need them archivally: store file vdev's on them. If you're just using them to replicate data to the DR site then you needn't even go as far as my workaround. I do agree that there's a missing tool: it's not possible to copy one subdirectory to another while preserving holes, forkey extended attributes, and ACL's. Also if Windows ACL's are going to be stored right in the filesystem, then Windows ACL's probably ought to be preserved over an rsync pipe between Solaris and EnTee, or a futuristic tarball written on one and extracted on the other. I don't agree that the missing tool is designed primarily for the narrow use-case of writing to ancient backup tapes: it's a more general tool. or, really, it's just a matter of documenting and committing the extra-OOB-gunk APIs and then fixing rsync and GNUtar. pgpk5wk2koJcZ.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20/01/2010 16:22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. Which isn't to say tape would be a better solution since it's going to be slower to restore etc. But it does show that tape can work out cheaper, especially since the cost of a high speed WAN link isn't required. JR You would also need to add at least one server to your library with fc cards. Then with most software you would need more tapes due to data fragmentation and a need to do regular full backups (with zfs+rsync you only do a full backup once). So in best case a library will cost about the same as disk based solution but generally will be less flexible, etc. If you would add any enterprise software on top of it (Legato, NetBackup, ...) then the price would change dramaticallly. Additionally with ZFS one could start using deduplication (in testing already). ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20/01/2010 17:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. Which isn't to say tape would be a better solution since it's going to be slower to restore etc. But it does show that tape can work out cheaper, especially since the cost of a high speed WAN link isn't required. JR You would also need to add at least one server to your library with fc cards. Then with most software you would need more tapes due to data fragmentation and a need to do regular full backups (with zfs+rsync you only do a full backup once). So in best case a library will cost about the same as disk based solution but generally will be less flexible, etc. If you would add any enterprise software on top of it (Legato, NetBackup, ...) then the price would change dramaticallly. Additionally with ZFS one could start using deduplication (in testing already). What I really mean is that a disk based solution used to be much more expensive than tapes but currently they are comparable in costs while often the disk based is more flexible. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 20, 2010, at 3:15 AM, Joerg Schilling wrote: Richard Elling richard.ell...@gmail.com wrote: ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. How quickly we forget ufsdump's limitations :-). For example, it is not supported for use on an active file system (known data corruption possibility) and UFS snapshots are, well, a poor hack and often not usable for backups. As the ufsdump(1m) manpage says, It seems you forgot that zfs also needs snapshots. There is nothing bad with snapshots. Yes, snapshots are a good thing. But most people who try fssnap on the UFS root file system will discover that it doesn't work; for reasons mentioned in the NOTES section of fssnap_ufs(1m). fssnap_ufs is simply a butt-ugly hack. So if you believe you can reliably use ufsdump to store a DR copy of root for a 7x24x365 production environment, then you probably believe the Backup Fairy will leave a coin under your pillow when your restore fails :-) Fortunately, ZFS snapshot do the right thing. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. A more apples to apples comparison would be to compare the storage only. Both removable drive and tape options require a server with FC or SCSI ports, so that can be excluded from the comparison. So for 30TB, assuming 2TB drives @ ~£100 with a pool built of 6 drive raidz vdevs 18 drives would be required plus 2 16 drive shelves. So each backup set would cost about £1800. So there's not a great deal of difference. With drives you also get the added benefit of keeping all your incrementals (as snapshots) on the archive set. HDD price per GB will continue to drop faster than tape, so it will be interesting to do the same comparison in 12 months. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g. ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz The ACL format used by Sun is undocumented.. man acltotext We are talking about TAR and I did give a pointer to the star archive format documentation, so it is obvious that I was talking about the ACL format from Sun tar. This format is not documented. It is, Sun's ZFS ACL aware tools use acltotext() to format ACLs. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
jr == Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk writes: jr While I am sure that star is technically a fine utility, the jr problem is that it is effectively an unsupported product. I have no problems with this whatsoever. jr If our customers find a bug in their backup that is caused by jr a failure in a Sun supplied utility, then they have a legal jr course of action. The customer's system administrators are jr covered because they were using tools provided by the jr vendor. The wrath of the customer would be upon Sun, not the jr supplier (us) or the supplier's technical lead (me). We were just talking about this somewhere else, actually: ``if something goes wrong, its their ass. but if nothing ever gets done, its nobody's fault.'' It's sad for me how much money is to be made supporting broken corporate cultures like that. I'm not saying you're wrong, just that you might not want to contribute to such a culture because you've chosen to endure it for a scratch. You need to have a better way to evaluate employees than micromanagement-by-the-clueless and vindictive hindsight. But the point that there's money to be made by bleeding it out of ossified broken American companies is well-taken. jr From the perspective of the business, the system administrator jr will have acted irresponsibly by choosing a tool that has no jr vendor support. From the perspective of MY business, I would much rather have the dark OOB acl/fork/whatever-magic that's gone into ZFS and NFSv4 supported in standard tools like rsync and GNUtar. This is, for example, what Apple achieved with CUPS and why I can share printers between Ubuntu and Mac OS effortlessly, and this increases the amount of money I'm willing to give Apple for their proprietary platform. The purpose of the tool I'm discussing definitely includes the same level of cooperation, so working with the existing best-in-class and most-popular tools, and reasonableness, might be better than brittle CYA support in some fringey '/opt/SUNWbkpkit/bin/VendorCP -Rf' tool. Even if you get your cyaCP tool you may find it doesn't achieve the ass-covering you wanted because these tools can be cheeky little bastards. Most of the other quirky little balkanized-platform Solaris-only tools are littered with straightjacketing assertions to avoid ``call generators'' and push the blame back onto the sysadmin, then there is some ``all bets are off'' flag to allowe you to actually accomplish job, like 'NOINUSECHECK=1 format -e'. Honestly...why bother playing this game? pgpBTS02xCBZW.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 20, 2010, at 12:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: [...] So you could provision a tape backup for just under £3 (~ $49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. [...] You would also need to add at least one server to your library with fc cards. Then with most software you would need more tapes due to data fragmentation and a need to do regular full backups (with zfs+rsync you only do a full backup once). So in best case a library will cost about the same as disk based solution but generally will be less flexible, etc. If you would add any enterprise software on top of it (Legato, NetBackup, ...) then the price would change dramaticallly. Additionally with ZFS one could start using deduplication (in testing already). Regardless of the economics of tape, nowadays you generally need to go to disk first because trying to stream at 120 MB/s (LTO-4) really isn't practical over the network, directly from the client. So in the end you'll be starting with disk (either DAS or VTL or whatever), and generally going to tape if you need to keep stuff that's older than (say) 3-6 months. Tape also doesn't rotate while it's sitting there, so if it's going to be sitting around for a while (e.g., seven years) better to use tape than something that sucks up power. LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB and (uncompressed) writes at 180 MB/s. The only way to realistically feed that is from disk. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Ian Collins i...@ianshome.com wrote: We are talking about TAR and I did give a pointer to the star archive format documentation, so it is obvious that I was talking about the ACL format from Sun tar. This format is not documented. It is, Sun's ZFS ACL aware tools use acltotext() to format ACLs. Please don't reply without checking facts. The fact that you know that there is salt in the soup does not give you the whole list of ingredients Please look into the Sun tar format to understand that you are wrong. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Miles Nordin car...@ivy.net wrote: From the perspective of MY business, I would much rather have the dark OOB acl/fork/whatever-magic that's gone into ZFS and NFSv4 supported in standard tools like rsync and GNUtar. This is, for example, what GNU tar does not support any platform speficic feature on any OS. Don't expect that GNU tar will ever add such properties.. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, Jan 20, 2010 at 2:52 PM, David Magda dma...@ee.ryerson.ca wrote: On Jan 20, 2010, at 12:21, Robert Milkowski wrote: On 20/01/2010 16:22, Julian Regel wrote: [...] So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. [...] You would also need to add at least one server to your library with fc cards. Then with most software you would need more tapes due to data fragmentation and a need to do regular full backups (with zfs+rsync you only do a full backup once). So in best case a library will cost about the same as disk based solution but generally will be less flexible, etc. If you would add any enterprise software on top of it (Legato, NetBackup, ...) then the price would change dramaticallly. Additionally with ZFS one could start using deduplication (in testing already). Regardless of the economics of tape, nowadays you generally need to go to disk first because trying to stream at 120 MB/s (LTO-4) really isn't practical over the network, directly from the client. I remember for about 5 years ago (before LT0-4 days) that streaming tape drives would go to great lengths to ensure that the drive kept streaming - because it took so much time to stop, backup and stream again. And one way the drive firmware accomplished that was to write blocks of zeros when there was no data available. This also occurred when the backup source was sending a bunch of small files, which took longer to stream and did'nt produce enough data to keep the drive writing useful data. And if you had the tape hardware setup to do compression, then, assuming a normal 2:1 compression ratio, you'd need to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec. The net result was the consumption of a lot more tape than a back-of-the-napkin calculation told you was required. Obviously at higher compression ratios or with the higher stream data write rates you quote below - this problem becomes more troublesome. So I agree with your conclusion: The only way to realistically feed that is from disk. So in the end you'll be starting with disk (either DAS or VTL or whatever), and generally going to tape if you need to keep stuff that's older than (say) 3-6 months. Tape also doesn't rotate while it's sitting there, so if it's going to be sitting around for a while (e.g., seven years) better to use tape than something that sucks up power. LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB and (uncompressed) writes at 180 MB/s. The only way to realistically feed that is from disk. ___ Regards, -- Al Hopper Logical Approach Inc,Plano,TX a...@logical-approach.com Voice: 972.379.2133 Timezone: US CDT OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007 http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 20 jan 2010, at 17.22, Julian Regel wrote: It is actually not that easy. Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO. Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare + 2x OS disks. The four raidz2 group form a single pool. This would provide well over 30TB of logical storage per each box. Now you rsync all the data from your clients to a dedicated filesystem per client, then create a snapshot. All snapshots are replicated to a 2nd x4540 so even if you would loose entire box/data for some reason you would still have a spare copy. Now compare it to a cost of a library, lto drives, tapes, software + licenses, support costs, ... See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html I've just read your presentation Robert. Interesting stuff. I've also just done a pen and paper exercise to see how much 30TB of tape would cost as a comparison to your disk based solution. Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 despite the Sun website saying it's a supported option. Each LTO3 has a native capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB native per library). To match the 30TB in your solution, we'd need two libraries totalling £28000. LTO3 has native capacity 400 GB, LTO4 has 800. The price is about the same per tape and per drive, a little higher for LTO4. You would also need 100 LTO3 tapes to provide 30TB of native storage. I recently bought a pack of 20 tapes for £340, so five packs would be £1700. Or rather 37 LTO4 tapes, and only one 48 tape library. But that doesn't matter, the interesting part is that one now can use whatever best solves the problem at hand. So you could provision a tape backup for just under £3 (~$49000). In comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price £30900. I've not factored in backup software since you could use an open source solution such as Amanda or Bacula. Which isn't to say tape would be a better solution since it's going to be slower to restore etc. But it does show that tape can work out cheaper, especially since the cost of a high speed WAN link isn't required. Reading from tape is normally faster than reading from (a single) disk. Seek time of course isn't. /ragge ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 21 jan 2010, at 00.20, Al Hopper wrote: I remember for about 5 years ago (before LT0-4 days) that streaming tape drives would go to great lengths to ensure that the drive kept streaming - because it took so much time to stop, backup and stream again. And one way the drive firmware accomplished that was to write blocks of zeros when there was no data available. This also occurred when the backup source was sending a bunch of small files, which took longer to stream and did'nt produce enough data to keep the drive writing useful data. And if you had the tape hardware setup to do compression, then, assuming a normal 2:1 compression ratio, you'd need to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec. The net result was the consumption of a lot more tape than a back-of-the-napkin calculation told you was required. Obviously at higher compression ratios or with the higher stream data write rates you quote below - this problem becomes more troublesome. So I agree with your conclusion: The only way to realistically feed that is from disk. Yes! Modern LTO drives can typically vary their speed about a factor four or so, so even if you can't keep up with the tape drive maximum speed, it will typically work pretty good anyway. If you can't keep up even then, it will have to stop, back up a bit, and restart, which will be _very_ slow. Having a disk system deliver data at 240 MB/s at the same time as you are writing to it can be a bit of a challenge. I haven't seen drives that fill out with zeros. Sounds like an ugly solution, but maybe it could be useful in some strange case. /ragge s ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Ragnar Sundblad ra...@csc.kth.se wrote: Yes! Modern LTO drives can typically vary their speed about a factor four or so, so even if you can't keep up with the tape drive maximum speed, it will typically work pretty good anyway. If you can't keep up even then, it will have to stop, back up a bit, and restart, which will be _very_ slow. Having a disk system deliver data at 240 MB/s at the same time as you are writing to it can be a bit of a challenge. And star implements a FIFO that is written in a way that dramatically reduces the sawtooth behavior seen typically with other applications. You just need to tell star to use a say 2 GB FIFO and star will be able to keep the tame streaming for a longer time before it waits until there is enough data for the next longer streaming period. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Thomas Burgess wonsl...@gmail.com wrote: so star is better than tar? Star is the oldest OSS tar implementation (it started in 1982). Star is (in contrary to Sun's tar and to GNU tar) able to create archives with attributes from recent POSIX standards and it implements aprox. twice as many features than GNU tar without forcing you to learn twice as many options ;-) Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Edward Ned Harvey sola...@nedharvey.com wrote: I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up, and do a stat on every file to see which files have changed since the last backup run. If you have a large filesystem, that can take a very long time. Star implements this in a very effective way (by using libfind) that is even faster that the find(1) implementation from Sun. The big advantage with star is that you get OS and filesystem independent archives that are compatible with recent POSIX standards and thus can be read on any POSIX.1-2001 compliant OS even if you don't have a star binary handy. I run incremental zfs send receive, and it completes typically in a minute or two. Did you make a test with star? Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Lassi Tuura l...@cern.ch wrote: I guess what I am after is, for data which really matters to its owners and which they actually had to recover, did people use tar/pax archives (~ file level standard archive format), dump/restore (~ semi-standard format based on files/inodes) or zfs send/receive (~ file system block level dump), or something else, and why? (Regardless of how these are implemented, hobby scripts or enterprise tools, how they dealt with possible media failure issues, etc.) star combines the advantages from ufsdump/ufsrestore (true incremental dumps) with the advantages of a POSIX standard achive format. Note that star is even at least 30% faster that ufsdump (although ufsdump reads the raw disk device while star uses the official OS filesystem interface). Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Miles Nordin car...@ivy.net wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. Star is such a tool. It privides all the features known from ufsdump but it is OS and filesystem indepentent. Once Sun shows interest in star, there will of course be also support for NTFS-ACLs. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Richard Elling richard.ell...@gmail.com wrote: OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead of /usr/bin, so gnu tar is the default. As of b130 (I'm not running an older build currently) the included gnu tar is version 1.22 which is the latest as released March 2009 at http://www.gnu.org/software/tar/ This causes Indiana to by default offer tar implementation that creates archives that are in conflict with the POSIX standard. Also note that GNU tar is unable to be used as a backup utility: - It does neither support ACLs not any other file attributes. - It failes even with simples modifications in the original filesystem and people will become aware of this problem at the time when they try to restore a backup that will not work in many cases. That is my understanding as well. It seems that the backup vendors are moving forward in a more-or-less vendor-specific way. This is not necessarily a bad thing, since there are open source solutions. But I see the requirements for backups being much more sophisticated than ufsdump was 25 years ago. hmmm... has ufsdump changed over the past 25 years? ;-) ufsdump did (slightly) change as it now allows you to specify a subdirectory instead of the whole filesystem. But what features are you missing? Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Daniel Carosone d...@geek.com.au wrote: I also don't recommend files 1Gb in size for DVD media, due to iso9660 limitations. I haven't used UDF enough to say much about any limitations there. ISO-9660 supports files up to 8 TB. Do you have a bigger pool? Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
+1 for zfsdump/zfsrestore Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. This has been something I've been working on for the last few days and I've not yet found an adequate solution. I've looked at tar, cpio and pax as potential tools for backing up a ZFS filesystem to tape, but each has limitations. As of today, Sun do not provide the ability to reliably backup a Solaris installation without resorting to third party tools. This is crazy! The beauty of ufsdump/ufsrestore is that because it's bundled with the operating system, I can perform bare metal recovery using a Solaris DVD and locally attached tape drive. It's simple and arguably essential for system administrators. From my perspective, Sun really need to create a zfsdump/zfsrestore set of commands that perform block level backups of a specified filesystem (or snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes (many of us still use and rely on tape!) etc. Does anyone know the process to formally request this (we have a support contract), or would I be wasting my time? Thanks JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 19, 2010, at 1:53 AM, Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. This has been something I've been working on for the last few days and I've not yet found an adequate solution. I've looked at tar, cpio and pax as potential tools for backing up a ZFS filesystem to tape, but each has limitations. As of today, Sun do not provide the ability to reliably backup a Solaris installation without resorting to third party tools. This is crazy! The beauty of ufsdump/ufsrestore is that because it's bundled with the operating system, I can perform bare metal recovery using a Solaris DVD and locally attached tape drive. It's simple and arguably essential for system administrators. Yep. And it was invented because there was no option other than tar at the time. Today, there are many very comprehensive backup solutions on the market including open source. Some are nicely integrated, such as NexentaStor and Zmanda. From my perspective, Sun really need to create a zfsdump/zfsrestore set of commands that perform block level backups of a specified filesystem (or snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes (many of us still use and rely on tape!) etc. In my crystal ball, I see a divergence away from traditional backup solution approaches. Today you can feel comfortable with backing up ZFS file systems because they provide a POSIX file system interface. Other datasets don't resemble POSIX file systems at all, and I see several different types of datasets with the opportunity to become popular with the basic ZVol getting a lot of attention lately. The win will go to the vendor who can provide the best value for the various datasets in the ZFS environment. Does anyone know the process to formally request this (we have a support contract), or would I be wasting my time? You can pile onto CR 5004379, want comprehensive backup strategy http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379 I can't answer the latter, but judging by the date of the CR, I wouldn't hold my breath. Give the fine folks at Zmanda a look. Also, the ADM project seems to be dead. Unfortunate? http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Tue, Jan 19, 2010 at 11:36 AM, Richard Elling richard.ell...@gmail.comwrote: On Jan 19, 2010, at 1:53 AM, Julian Regel wrote: When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. This has been something I've been working on for the last few days and I've not yet found an adequate solution. I've looked at tar, cpio and pax as potential tools for backing up a ZFS filesystem to tape, but each has limitations. As of today, Sun do not provide the ability to reliably backup a Solaris installation without resorting to third party tools. This is crazy! The beauty of ufsdump/ufsrestore is that because it's bundled with the operating system, I can perform bare metal recovery using a Solaris DVD and locally attached tape drive. It's simple and arguably essential for system administrators. Yep. And it was invented because there was no option other than tar at the time. Today, there are many very comprehensive backup solutions on the market including open source. Some are nicely integrated, such as NexentaStor and Zmanda. From my perspective, Sun really need to create a zfsdump/zfsrestore set of commands that perform block level backups of a specified filesystem (or snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes (many of us still use and rely on tape!) etc. In my crystal ball, I see a divergence away from traditional backup solution approaches. Today you can feel comfortable with backing up ZFS file systems because they provide a POSIX file system interface. Other datasets don't resemble POSIX file systems at all, and I see several different types of datasets with the opportunity to become popular with the basic ZVol getting a lot of attention lately. The win will go to the vendor who can provide the best value for the various datasets in the ZFS environment. Does anyone know the process to formally request this (we have a support contract), or would I be wasting my time? You can pile onto CR 5004379, want comprehensive backup strategy http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379 I can't answer the latter, but judging by the date of the CR, I wouldn't hold my breath. Give the fine folks at Zmanda a look. Also, the ADM project seems to be dead. Unfortunate? http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM -- richard I'm likely missing something, so please, fill me in. Aren't they simply asking for the functionality already provided by NDMP? -- --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
The beauty of ufsdump/ufsrestore is that because it's bundled with the operating system, I can perform bare metal recovery using a Solaris DVD and locally attached tape drive. It's simple and arguably essential for system administrators. Yep. And it was invented because there was no option other than tar at the time. Today, there are many very comprehensive backup solutions on the market including open source. Some are nicely integrated, such as NexentaStor and Zmanda. When I look at the documentation for Zmanda (http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS), it states that the command used to backup a ZFS filesystem depends on whether backing up ACLs are required (the default is not to(!)). If backing up ACLs are required - which they are for us - then the native /usr/bin/tar command is used. Given that /usr/bin/tar doesn't handle sparse files correctly and updates atime when creating archives, it's not possible to create a complete copy of a filesystem without making intrusive changes to the structure of the data and/or metadata. So it's arguable that ufsdump was a significant improvement over tar, and that the current archiving solutions for ZFS are actually a step backwards. From my perspective, Sun really need to create a zfsdump/zfsrestore set of commands that perform block level backups of a specified filesystem (or snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes (many of us still use and rely on tape!) etc. In my crystal ball, I see a divergence away from traditional backup solution approaches. Today you can feel comfortable with backing up ZFS file systems because they provide a POSIX file system interface. Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. JR ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. In the past few years growth in hard drive capacities has outstripped tapes to the extent that removable hard drives and ZFS snapshots have become a more cost effective and convenient backup media. What do people with many tens of TB use for backup these days? -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Julian Regel jrmailgate-zfsdisc...@yahoo.co.uk wrote: When I look at the documentation for Zmanda (http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS), it states that the command used to backup a ZFS filesystem depends on whether backing up ACLs are required (the default is not to(!)). If backing up ACLs are required - which they are for us - then the native /usr/bin/tar command is used. Given that /usr/bin/tar doesn't handle sparse files correctly and updates atime when creating archives, it's not possible to create a complete copy of a filesystem without making intrusive changes to the structure of the data and/or metadata. So it's arguable that ufsdump was a significant improvement over tar, and that the current archiving solutions for ZFS are actually a step backwards. From my perspective, Sun really need to create a zfsdump/zfsrestore set of commands that perform block level backups of a specified filesystem (or snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes (many of us still use and rely on tape!) etc. Sun's tar is not able to archive files 8 GB in a way that can be read on any other OS unless you use star. This is because Sun's tar does not implement support for the POSIX.1-2001 extended tar format. Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot understand them and probably never will be able to understand this format as it is not well defined for a portable program like star. Star supports to do backups without affecting atime (on Solaris) since 15 years already and star supports the withdrawn POSIX draft ACLs in a OS independent way. This allows to use star for platform independent ACL support in case you are using UFS on Solaris. Star will in future support NTFS ACLs in a OS independent way. If someone likes to contribute, he is of course welcome. As I am currently working on cdrtools-3.0-final, star is currently on maintenance only. What we need for the future is a OS/FS independent program that implements the needed features. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: Julian Regel wrote: Based on what I've seen in other comments, you might be right. Unfortunately, I don't feel comfortable backing up ZFS filesystems because the tools aren't there to do it (built into the operating system or using Zmanda/Amanda). Commercial backup solutions are available for ZFS. On what archiving programs are these solutions based on? I'm sure they use their own formats. The only one I have crossed paths with is NetVault. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Wed, 20 Jan 2010, Ian Collins wrote: Commercial backup solutions are available for ZFS. I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. True, but I wonder how viable its future is. One of my clients requires 17 LT04 types for a full backup, which cost more and takes up more space than the equivalent in removable hard drives. In the past few years growth in hard drive capacities has outstripped tapes to the extent that removable hard drives and ZFS snapshots have become a more cost effective and convenient backup media. In all of these discussions, people seem to forget some very important criteria. That important criteria is the time required for a full recovery from backup. If the company is not able to survive more than a week with total disruption of business, but the full restore will take three weeks, then the backup has been rendered 100% ineffective. When planning for recovery from disaster, the time to recover is exceedingly important. When using the backup to filesystem approach, that backup could be done to a remote mirrored system (filesystems + hardware) which is sufficiently distant to protect against any local disaster. If there is a disaster in the local data center, that system could be immediately put on line (assuming adequate connectivity), or that system could be loaded on a truck for overnight delivery as a replacement to the data center. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Ian Collins i...@ianshome.com wrote: Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot understand them and probably never will be able to understand this format as it is not well defined for a portable program like star. Is that because they are NFSv4 ACLs? tar uses the formatting routines from libacl to archive them. The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g. ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz The ACL format used by Sun is undocumented.. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
jk == Jerry K sun.mail.lis...@oryx.cc writes: jk +1 jk for zfsdump/zfsrestore -1 I don't think a replacement for the ufsdump/ufsrestore tool is really needed. From now on, backups just go into Another Zpool. We need the zfs send stream format commitment (stream format depends only on zfs version, not on zpool version or kernel version), but that's enough. If people are really still backing up to tapes or DVD's, just use file vdev's, export the pool, and then copy the unmounted vdev onto the tape or DVD. The requirement that your backup be staged on a disk isn't an obstacle in the backup direction: Amanda already has this requirement. Amanda, when I used it, did *not* have this requirement in the restore direction so one would have to keep that in mind: if using tapes, he needs a scratch disk as big as the biggest tape file on the DR site or the development environment or Compliance Extractor Station or wherever the restore is happening. File vdev's have all the wanted characteristics: bitflip resiliency, checksums, ability to represent all the opaque mysteryfeatures of inner ZFS's, snapshot/clone efficiency, and importability by future ZFS kernels ~forever. It still has the all-or-nothing-restore downside, but maybe we can live with that. Those who can't might stick to spinning-disk backups. It might be nice to have a ``read-only vdev'' like a mac os compressed disk image that can occupy the same pool next to a read-write vdev---this would be neat for livecd's and incrementals---but it's a neat feature, not something blocking a frequent/unavoidable sysadmin task. What's needed IMHO is non-ZFS-specific tools like tar/pax/cpio/rsync that understand all this new ACL and opaque-fork garbage that filesystems are growing (and not just Sun's either). It's needed to copy between NFSv4 and ZFS, and also to split/combine subdirectories into separate/common ZFS filesystems without losing any of the newfangled mysterybaggage. It has as much to do with shaking corner cases and awkwardnesses out of the newfangled API's as it does with actually accomplishing a sysadmin task: the best documentation for the weird appendages in various modern Unix filesystems would probably be the tool itself, if we had it. Without this kind of tool and the API shakeout that making it would require, our systems are slowly turning into shitty Windows boxes that need some black magic proprietary tool like Ghost or PQDI to capture all the silly out-of-band garbage their installations depend on and/or accumulate. pgpnDeqTYeOuV.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
ic == Ian Collins i...@ianshome.com writes: I know tape backup isn't sexy, but it's a reality for many of us and it's not going away anytime soon. ic True, but I wonder how viable its future is. One of my ic clients requires 17 LT04 types for a full backup, which cost ic more and takes up more space than the but Ian, aiui SOX requires backups onto a write-once non-eraseable medium, because of all the discovery shennanigans and claimed-incompetence around the time of the Worldcomm and Enron bankruptcy near the start of Bush II. The law in practice is sort of like a tiger that chases the herd and eats the sick animal that's lagging behind, and currently I think the herd is using ``WORM tape'' which is a tape that's made to appear to be append-only by the tape drive's firmware. These tapes exist now and must be retained for seven years, so even if you invented a newfangled gadget meeting the WORM requirements TODAY so that you can stay in the middle of the herd without using tape, you still will not get rid of tape for seven years. pgp2HKJuL5cLI.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Tue, Jan 19, 2010 at 12:16:01PM +0100, Joerg Schilling wrote: Daniel Carosone d...@geek.com.au wrote: I also don't recommend files 1Gb in size for DVD media, due to iso9660 limitations. I haven't used UDF enough to say much about any limitations there. ISO-9660 supports files up to 8 TB. However, some implementations have (or had?) limitations which meant that they couldn't read such disks. Some of those platforms even had problems (or required remembering arcane options and flags for large file support) when copying these files back to hard disk. Linux was a prime culprit, though not the only one. I didn't use linux, but I never knew what might be at hand when needing to read disks for recovery, so I wanted them to be portable. I envisaged using multiple machines to read more disks in parallel, for example. It may not be as relevant for this use case, nor perhaps for more modern platforms, but it was a habit I developed earlier for other kinds of archives. I apologise, though, for misattributing this limitation to the filesystem spec. I misremembered the reason why I chose the limit. It was a practical limit, and the practicalities may well have changed since. Do you have a bigger pool? I certainly don't have a bigger DVD media :-) (Yes, I know iso9660 also has multivolume support, but I'm similarly wary of being able to read it back on all platforms, since it's not encountered often, and there's no need when the source files can be split to better effect.) -- Dan. pgpqa8vKafJ1t.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
There is a tendency to conflate backup and archive, both generally and in this thread. They have different requirements. Backups should enable quick restore of a full operating image with all the necessary system level attributes. They concerned with SLA and uptime and outage and data loss window criteria. Archives are usually much more concerned with application data, which may have been specially prepared for the future user of the archive. Portability is often more important here, both in archive format but also in data schema within, even at the expense of some kinds of technical integrity or completeness (e.g. ACLs). Adding regulatory requirements into the mix further complicates matters. If one solution can address the union of all requirements, then you're in luck, but that's not always the case. Sometimes the best way to resolve the tension is to save the data differently for different purposes. -- Dan. pgpfsQPyUEOVE.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Message: 3 Date: Tue, 19 Jan 2010 15:48:52 -0500 From: Miles Nordin car...@ivy.net To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability? Message-ID: oqpr55lt1n@castrovalva.ivy.net Content-Type: text/plain; charset=us-ascii I don't think a replacement for the ufsdump/ufsrestore tool is really needed. From now on, backups just go into Another Zpool. We need the zfs send stream format commitment (stream format depends only on zfs version, not on zpool version or kernel version), but that's enough. If people are really still backing up to tapes or DVD's, just use file vdev's, export the pool, and then copy the unmounted vdev onto the tape or DVD. The requirement that your backup be staged on a disk isn't an obstacle in the backup direction: Amanda already has this requirement. Amanda, when I used it, did *not* have this requirement in the restore direction so one would have to keep that in mind: if using tapes, he needs a scratch disk as big as the biggest tape file on the DR site or the development environment or Compliance Extractor Station or wherever the restore is happening. While this idea may be fine for a home user…those us of who have customers in the enterprise still have a need for tape backups. And some of those enterprises require backup mechanism that can be easily used in a DR situation. ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. For example, one customer, local government, requires a backup that can be taken offsite and used in a DR situation. They have 2 sun servers. They have very little Solaris knowledge. So whatever I provide them has to be easy and easily documented. Lack of zfsdump/restore means I either have to forgo moving them to ZFS root, or have to add in a third party application…like I'm really going to meet their requirements with Amanda or Backula… This lack in Solaris is just plain ludicrous to at least some parts of the business/enterprise world. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 19, 2010, at 4:26 PM, Allen Eastwood wrote: Message: 3 Date: Tue, 19 Jan 2010 15:48:52 -0500 From: Miles Nordin car...@ivy.net To: zfs-discuss@opensolaris.org Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability? Message-ID: oqpr55lt1n@castrovalva.ivy.net Content-Type: text/plain; charset=us-ascii I don't think a replacement for the ufsdump/ufsrestore tool is really needed. From now on, backups just go into Another Zpool. We need the zfs send stream format commitment (stream format depends only on zfs version, not on zpool version or kernel version), but that's enough. If people are really still backing up to tapes or DVD's, just use file vdev's, export the pool, and then copy the unmounted vdev onto the tape or DVD. The requirement that your backup be staged on a disk isn't an obstacle in the backup direction: Amanda already has this requirement. Amanda, when I used it, did *not* have this requirement in the restore direction so one would have to keep that in mind: if using tapes, he needs a scratch disk as big as the biggest tape file on the DR site or the development environment or Compliance Extractor Station or wherever the restore is happening. While this idea may be fine for a home user…those us of who have customers in the enterprise still have a need for tape backups. And some of those enterprises require backup mechanism that can be easily used in a DR situation. ufsdump/restore was perfect in that regard. The lack of equivalent functionality is a big problem for the situations where this functionality is a business requirement. How quickly we forget ufsdump's limitations :-). For example, it is not supported for use on an active file system (known data corruption possibility) and UFS snapshots are, well, a poor hack and often not usable for backups. As the ufsdump(1m) manpage says, The ufsdump command can only be used on unmounted file sys- tems, or those mounted read-only. Attempting to dump a mounted, read-write file system might result in a system disruption or the inability to restore files from the dump. Consider using the fssnap(1M) command to create a file sys- tem snapshot if you need a point-in-time image of a file system that is mounted. So, if you are taking ufsdumps of an active file system for business requirements, then perhaps you should rethink the solution. For example, one customer, local government, requires a backup that can be taken offsite and used in a DR situation. They have 2 sun servers. They have very little Solaris knowledge. So whatever I provide them has to be easy and easily documented. Lack of zfsdump/restore means I either have to forgo moving them to ZFS root, or have to add in a third party application…like I'm really going to meet their requirements with Amanda or Backula… Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup solutions on snapshots, I assert the problem is low priority. This lack in Solaris is just plain ludicrous to at least some parts of the business/enterprise world. Methinks you never read the ufsdump man page? :-) -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup solutions on snapshots, I assert the problem is low priority. What I have issue with is the idea that no one uses/should use tape any more. There are places for tape and it still has value as a backup device. In many cases in the past, ufsdump, despite it's many issues, was able to restore working OS's, or individual files. Perfect, not by a long shot. But it did get the job done. As was pointed out earlier, all I needed was a Solaris CD (or network boot) and I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted, same thing…and I can restore just the file(s) I need. And while it's been a few years since I've read the man page on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing with a downed system. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
I use zfs send/recv in the enterprise and in smaller environments all time and it's is excellent. Have a look at how awesome the functionally is in this example. http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs Regards, Mike -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Joerg Schilling wrote: Ian Collins i...@ianshome.com wrote: Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot understand them and probably never will be able to understand this format as it is not well defined for a portable program like star. Is that because they are NFSv4 ACLs? tar uses the formatting routines from libacl to archive them. The correct way to archivbe ACLs would be to put them into extended POSIX tar attrubutes as star does. See http://cdrecord.berlios.de/private/man/star/star.4.html for a format documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g. ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz The ACL format used by Sun is undocumented.. man acltotext -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup solutions on snapshots, I assert the problem is low priority. What I have issue with is the idea that no one uses/should use tape any more. There are places for tape and it still has value as a backup device. In many cases in the past, ufsdump, despite it's many issues, was able to restore working OS's, or individual files. Perfect, not by a long shot. But it did get the job done. As was pointed out earlier, all I needed was a Solaris CD (or network boot) and I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted, same thing…and I can restore just the file(s) I need. And while it's been a few years since I've read the man page on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing with a downed system. For a full recovery, you can archive a send stream and receive it back. With ZFS snapshots, the need for individual file recovery from tape is much reduced. The backup server I manage for a large client has 60 days of snaps and I can't remember when they had to go to tape to recover a file. -- Ian. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Star implements this in a very effective way (by using libfind) that is even faster that the find(1) implementation from Sun. Even if I just find my filesystem, it will run for 7 hours. But zfs can create my whole incremental snapshot in a minute or two. There is no way star or any other user-space utility that walks the filesystem can come remotely close to this performance. Such performance can only be implemented at the filesystem level, or lower. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 19, 2010, at 22:54 , Ian Collins wrote: Allen Eastwood wrote: On Jan 19, 2010, at 18:48 , Richard Elling wrote: Many people use send/recv or AVS for disaster recovery on the inexpensive side. Obviously, enterprise backup systems also provide DR capabilities. Since ZFS has snapshots that actually work, and you can use send/receive or other backup solutions on snapshots, I assert the problem is low priority. What I have issue with is the idea that no one uses/should use tape any more. There are places for tape and it still has value as a backup device. In many cases in the past, ufsdump, despite it's many issues, was able to restore working OS's, or individual files. Perfect, not by a long shot. But it did get the job done. As was pointed out earlier, all I needed was a Solaris CD (or network boot) and I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted, same thing…and I can restore just the file(s) I need. And while it's been a few years since I've read the man page on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing with a downed system. For a full recovery, you can archive a send stream and receive it back. With ZFS snapshots, the need for individual file recovery from tape is much reduced. The backup server I manage for a large client has 60 days of snaps and I can't remember when they had to go to tape to recover a file. -- Ian. Let's see… For full recovery, I have to zfs send to something, preferably that understands tape (yes, I know I can send to tape directly, but how well does zfs send handle the end of the tape? auto-changers?). Then for individual file recovery, I have snaphots…which I also have to get on to tape…if I want to have them available on something other than the boot devices. Now…to recover the entire OS, perhaps not so bad…but that's one tool. And to recover the one file, say a messed up /etc/system, that's preventing my OS from booting? Have to get that snapshot where I can use it first…oh and restoring individual files and not the entire snapshot? At best, it's an unwieldy process. But does it offer the simplicity that ufsdump/ufsrestore (or dump/restore on how many Unix variants…) did? No way. I suppose it might be fair to say, strictly speaking, that backup/restore probably should be dealt with as a ZFS issue. It's kind of ironic that Microsoft offers a backup utility and Sun is basically dropping theirs. It might be better to frame the debate somewhere else, but zfs send/receive and snapshots are not the same as a built-in OS utility for backing up and restoring the OS and OS files. A simple, effective dump/restore that deals with all the supported file systems, can deal with tape or disk, and allows for complete OS restore or individual file restore, and can be run from a install CD/DVD. As much as I love ZFS and as many problems as it does solve, leaving this out was a mistake, IMO. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up, and do a stat on every file to see which files have changed since the last backup run. If you have a large filesystem, that can take a very long time. I recently switched to ZFS specifically for this reason. Previously, I was doing a nightly rsync on 1Tb of data. It required 10 hrs every night to run, and copy typically a few hundred megs that had changed that day. Now, I run incremental zfs send receive, and it completes typically in a minute or two. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. That's a pretty cool idea. Then you've still got the entire zfs volume inside of a file, but you're able to mount and extract individual files if you want, and you're able to pipe your zfs send directly to zfs receive, and basically you get all the benefits of zfs send receive, but also get all the benefits of being able to treat your backup like a file or a set of files. Pretty cool. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in files with very few blocks). However, I don't see why further ZFS perfomance work couldn't close that gap, since rsync will always need to compare directories and timestamps. Phil On 18 Jan 2010, at 08:07, Edward Ned Harvey sola...@nedharvey.com wrote: I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up, and do a stat on every file to see which files have changed since the last backup run. If you have a large filesystem, that can take a very long time. I recently switched to ZFS specifically for this reason. Previously, I was doing a nightly rsync on 1Tb of data. It required 10 hrs every night to run, and copy typically a few hundred megs that had changed that day. Now, I run incremental zfs send receive, and it completes typically in a minute or two. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Mon, Jan 18, 2010 at 3:59 AM, Phil Harman phil.har...@gmail.com wrote: YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in files with very few blocks). However, I don't see why further ZFS perfomance work couldn't close that gap, since rsync will always need to compare directories and timestamps. Phil The best info i've read on this was on this blog: http://richardelling.blogspot.com/2009/01/parallel-zfs-sendreceive.html On 18 Jan 2010, at 08:07, Edward Ned Harvey sola...@nedharvey.com wrote: I still believe that a set of compressed incremental star archives give you more features. Big difference there is that in order to create an incremental star archive, star has to walk the whole filesystem or folder that's getting backed up, and do a stat on every file to see which files have changed since the last backup run. If you have a large filesystem, that can take a very long time. I recently switched to ZFS specifically for this reason. Previously, I was doing a nightly rsync on 1Tb of data. It required 10 hrs every night to run, and copy typically a few hundred megs that had changed that day. Now, I run incremental zfs send receive, and it completes typically in a minute or two. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 18/01/2010 08:59, Phil Harman wrote: YMMV. At a recent LOSUG meeting we were told of a case where rsync was faster than an incremental zfs send/recv. But I think that was for a mail server with many tiny files (i.e. changed blocks are very easy to find in files with very few blocks). After changes around build 114 (iirc) this shouldn't be the case anymore znd an incremental zfs send should always be able to at least match the performance of incremental rsync. Now with lots of files where only a small subset of them changes incremental zfs send should be much faster and that's my observation. -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
or you might do something like: http://milek.blogspot.com/2009/12/my-presentation-at-losug.html however in your case if all your clients are running zfs only filesystems then relaying just on zfs send|recv might be a good idea. -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Hi, .. it's hard to beat the convenience of a backup file format, for all sorts of reasons, including media handling, integration with other services, and network convenience. Yes. Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. This is an interesting suggestion :-) Did I understand you correctly that once a slice is written, zfs won't rewrite it? In other words, I can have an infinitely growing pile of these slices, and once zfs fills one file up (or one raidz2 set of files), I flush it to tape/optical disk/whatever, and zfs won't change it ever after? When I need more space, I just add more slices, but old slices are effectively read-only? Or did I misunderstand how you meant it work? It sounded very interesting but my understanding on zfs is currently limited :-) And perhaps most importantly: has anyone actually done this for their back-ups and has success stories on restore after media failure? Regards, Lassi ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
mg == Mike Gerdts mger...@gmail.com writes: tt == Toby Thain t...@telegraphics.com.au writes: tb == Thomas Burgess wonsl...@gmail.com writes: mg Yet it is used in ZFS flash archives on Solaris 10 and are mg slated for use in the successor to flash archives. in FLAR, ``if a single bit's flipped, the whole archive and any incrementals descended from it are toast'' is a desireable characteristic. FLAR is a replication tool, just like 'zfs send | zfs receive' together are, and the sensitivity's desireable because you want a deterministicaly exact replica. If zpools themselves had the characteristic, ``single bit flipped, entire pool lost,'' few would be happy with that. In fact it's part of the ZFS kool-aid cocktail that bitflips are detected and reported, and that critical metadata is written several times far apart on the pool so the damage of a single bitflip, even on an unredundant pool, is limited. Other things that aren't ``enterprise backup solutions'', like tarballs, are also resilient to single bit flips. so, why tolerate this fragility from a backup format? It's not appropriate. The other problems are lack of a verification tool that doesn't involve ponderous amounts of kernel code revision-bound to hardware drivers and such that may one day become unobtainable, the possibility of painting yourself into a corner (ex. if you are backing up a 17TB filesystem, you must provide 17TB of restore space or else there is no way to access a 2kB file), and the stream format which is intentionally tied to the relatively often-changing zfs version so that it can make exact copies but at the cost of constraining the restore environment which may be separated from the backup environment by seven years and/or in the midst of a disaster in a way that tarballs and cpios and so on don't constrain. Another problem is that the snv_112 man page says this: -8- The format of the stream is evolving. No backwards com- patibility is guaranteed. You may not be able to receive your streams on future versions of ZFS. -8- I think the new man page says something more generous. The use case here is, we should be able to: old solarisnew solaris newer solaris zfs send - zfs recv \ \_ zpool upgrade -- (not 'zfs upgrade') -- zfs send | - zfs recv zfs recv - zfs send That is, the stream format should be totally deterministic given the zfs version, and not depend on the zpool version nor the kernel/userland version. This way a single backup pool can hold the backups of several different versions of solaris. The alternative to this level of guarantee is for your archival backup to involve a ball of executable code like a LiveCD or a .VDI, and that's just not on. Seven to ten years, people---there are laws TODAY that require archiving WORM media that long, and being able to read it, which means no rewriting for a decade. That's a ``hard requirement'' as Christopher says, _today,_ never mind what's desireable or useful. I'm not sure the new man page makes a promise quite that big as accomodating the above chart, but IIRC it's much better than the old one. anyway, zfs send and receive are very good replication tools, and replication can be used for backup (by replicating into another zpool *NOT* by storing the stream modulo the caveat above) or for system install (by recv'ing multiple times like FLAR). If you choose to store zfs send streams as backups, the least you can do is warn those you advise that zfs send streams are different from other kidns of backup stream, because sysadmin experience in how to write backups is old and hard-won, and these tools diverge from it. they diverge usefully---I'm not putting them down---I'm saying *don't use them that way* unless you are able to think through $subtleproblems, after which you'll probably not want to use them that way. tt I can see the temptation, but isn't it a bit under-designed? I tt think Mr Nordin might have ranted about this in the past... no, I think it's great for FLAR. single-bit-self-destruct is exactly what's wanted for FLAR. for those case where you could somehow magically capture and replay an rsync stream it's great. It's not a dumb design because IMHO a single format probably can't do well for both replication and backup, but it's not a backup format, enterprise or otherwise. What escalates my objection into a rant is the way ``enterprise backup solution'' gets tossed around as some kind of soundbite hatetorial prole phrase substituting and blocking off exploring the actual use cases which we've done over and
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Jan 18, 2010, at 11:04 AM, Miles Nordin wrote: ... Another problem is that the snv_112 man page says this: -8- The format of the stream is evolving. No backwards com- patibility is guaranteed. You may not be able to receive your streams on future versions of ZFS. -8- I think the new man page says something more generous. The Solaris 10 10/09 zfs(1m) man page says: The format of the stream is committed. You will be able to receive your streams on future versions of ZFS. I'm not sure when that hit snv, but obviously it was after b112. ... the solaris userland is ancient, so if you use gtar you'll be surprised less often, like IMHO you are less likely to have silently truncated files, but then it's maintained upstream so it's missing support for all these forked files, mandatory labels, and windows ACL's that they're piling into ZFS now for NAS features. OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead of /usr/bin, so gnu tar is the default. As of b130 (I'm not running an older build currently) the included gnu tar is version 1.22 which is the latest as released March 2009 at http://www.gnu.org/software/tar/ When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. That is my understanding as well. It seems that the backup vendors are moving forward in a more-or-less vendor-specific way. This is not necessarily a bad thing, since there are open source solutions. But I see the requirements for backups being much more sophisticated than ufsdump was 25 years ago. hmmm... has ufsdump changed over the past 25 years? ;-) -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Mon, Jan 18, 2010 at 07:34:51PM +0100, Lassi Tuura wrote: Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. This is an interesting suggestion :-) Did I understand you correctly that once a slice is written, zfs won't rewrite it? In other words, I can have an infinitely growing pile of these slices, and once zfs fills one file up (or one raidz2 set of files), I flush it to tape/optical disk/whatever, and zfs won't change it ever after? When I need more space, I just add more slices, but old slices are effectively read-only? No. The intention of this scheme is to have an ongoing backup pool, with a rolling set of historical snapshots collected together. I wouldn't suggest growing the pool indefinately, because you'll need an indefinite number of pool backing files online when it comes time for a distant-future restore. Even if you were never to delete old snapshots, zfs will still update some of the metadata in each file, or raidz set of files, whenever you make changes to your backup pool. You will need to sync each of these files to your backup media or offsite storage, to have a consistent pool to restore from. However, you can add new top-level vdevs (e.g. raidz set of files) as your backup pool starts to fill. Until you start to recycle space (by deleting old snapshots from the backup pool) there won't be much new data written to the original full files. Therefore, if your offsite replication of these files is efficient for incremental changes (e.g. rsync) then they will update quickly. The same will happen for changes within the existing files, of course, if they're not yet full or if you are recycling space. It's the same data, all that changes is how it's spread among the files. If you're writing to tapes or dvd's, you'll either need to rewrite the files completely every time, or layer some *other* incremental mechanism on top (perhaps your existing large-scale enterprise VTL, which you'd like to continue using, or are forced to continue using :). I don't mind rewriting tape media every time.. keep 2 or three sets and just roll between them each time. There are advantages to having a predictable cycle - knowing that you're going to need exactly N tapes this week and they will take T time to write out. You also get the benefit of all other D-to-D-to-T schemes, that this can be done at leisure asynchronously to host backup windows. For DVD's its a little more annoying, but at least the media is cheap. For these purposes, you can also consider removable hard disks as tapes. As I replace one generation of drives with the next, higher capacity, I intend for the previous ones can move to backup service. And perhaps most importantly: has anyone actually done this for their back-ups and has success stories on restore after media failure? For me it's somewhere between concept and regular practice. It works, I've tinkered with it and done intermittent runs at archiving off the pool files and test imports and restores, but not written automation or made it as regular practice as I should. I've used it to drop backups of my opensolaris hosts onto enterprise backed-up servers, in non-solaris customer environments. There are some wrinkles, like you can't mount zpool files off read-only DVD media directly - you have to copy them to r/w scratch space because import wants to update metadata (is there an RFE for read-only import? I forget). This is mildly irritating at worst - copying one or two dvd's worth of files is easy, yet any more than this in your backup pool, you'll need to copy anyway for lack of many dvd readers at once. I also don't recommend files 1Gb in size for DVD media, due to iso9660 limitations. I haven't used UDF enough to say much about any limitations there. -- Dan. pgpwXMKusfxKP.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 18/01/2010 18:28, Lassi Tuura wrote: Hi, Here is the big difference. For a professional backup people still typically use tapes although tapes have become expensive. I still believe that a set of compressed incremental star archives give you more features. Thanks for your comments! I think I roughly understand the feature trade-offs, and have some experience with back-up solutions ranging from simple to enterprise. I guess what I am after is, for data which really matters to its owners and which they actually had to recover, did people use tar/pax archives (~ file level standard archive format), dump/restore (~ semi-standard format based on files/inodes) or zfs send/receive (~ file system block level dump), or something else, and why? (Regardless of how these are implemented, hobby scripts or enterprise tools, how they dealt with possible media failure issues, etc.) Other than the file system vs. file restore, is there any concern in doing the block level thing? Concerns as in mind being in the line of fire if it fails? :-) What we are doing basically is: 1. incremental rsync from a client to a dedicated filesystem for that client 2. snapshot after rsync finished 3. go to #1 for next backup a pool is a dynamic stripe across raidz-2 groups + hot spares. Then selected clients along with all their backups (snapshots) are replicated to another device which is exactly the same hardware/software configuration. Add to it a management of snapshots (retention policies), reporting, etc. and you have a pretty good backup solution which allows you to restore a single file or entire filesystem with a very easy access to any backup. And it works for different OSes as clients. See more details at http://milek.blogspot.com/2009/12/my-presentation-at-losug.html -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Mon, Jan 18, 2010 at 01:38:16PM -0800, Richard Elling wrote: The Solaris 10 10/09 zfs(1m) man page says: The format of the stream is committed. You will be able to receive your streams on future versions of ZFS. I'm not sure when that hit snv, but obviously it was after b112. ... I only noticed it recently. My guess was that it arrived together with the format changes for zfs send -D. As handy as that change is, it doesn't address some of the other deficiencies involved in trying to repurpose a replication stream as an archive format. When we brought it up last time, I think we found no one knows of a userland tool similar to 'ufsdump' that's capable of serializing a ZFS along with holes, large files, ``attribute'' forks, windows ACL's, and checksums of its own, and then restoring the stream in a filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'. It was precisely the realisation that the zpool-in-a-file format had all of the desired characteristics that led to the scheme I described elsewhere in this thread. (It wasn't one of my criteria, but this goes up to and including the userland tool component - although I've not tried, I understand there are userland versions of the zfs tools in the test suite.) It has the advantage of being entirely common with the original, so it will be well tested and keep in sync with new features (dedup, crypto, next+N). A focus on formalising this usage pattern would bring further benefits in terms of features (e.g, as a worthwhile use case for read only import) and of practices (documentation and process). -- Dan. pgpmzgIoI2655.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
NO, zfs send is not a backup. Understood, but perhaps you didn't read my whole message. Here, I will spell out the whole discussion: If you zfs send somefile it is well understood there are two big problems with this method of backup. #1 If a single bit error is introduced into the file, then the whole data stream is corrupt. #2 If you want to restore just a subset of the filesystem, you cannot. The only option available is to restore the whole filesystem. Instead, it is far preferable to zfs send | zfs receive ... That is, receive the data stream on external media as soon as you send it. By receiving the data stream onto external media, instead of just saving the datastream as a file on external media ... You solve both of the above problems. Obviously this is only possible with external disks, and not possible with tapes. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Toby Thain t...@telegraphics.com.au wrote: Yet it is used in ZFS flash archives on Solaris 10 I can see the temptation, but isn't it a bit under-designed? I think Mr Nordin might have ranted about this in the past... Isn't flash cpio based and thus not prepared for the future? Cpio coes not support sparse files and is unable to archive files 8 GB. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Edward Ned Harvey sola...@nedharvey.com wrote: NO, zfs send is not a backup. Understood, but perhaps you didn't read my whole message. Here, I will spell out the whole discussion: ... Instead, it is far preferable to zfs send | zfs receive ... That is, receive the data stream on external media as soon as you send it. By receiving the data stream onto external media, instead of just saving the datastream as a file on external media ... You solve both of the above problems. Obviously this is only possible with external disks, and not possible with tapes. Here is the big difference. For a professional backup people still typically use tapes although tapes have become expensive. I still believe that a set of compressed incremental star archives give you more features. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Cpio coes not support sparse files and is unable to archive files 8 GB. Jörg I found this out the hard way last time i used it. I was backing up all my data from one system to another using cpio and i had a bunch of movies over 8GB (720p and 1080p mkv files) none of them worked. I never use cpio anymore. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Sun, Jan 17, 2010 at 05:31:39AM -0500, Edward Ned Harvey wrote: Instead, it is far preferable to zfs send | zfs receive ... That is, receive the data stream on external media as soon as you send it. Agree 100% - but.. .. it's hard to beat the convenience of a backup file format, for all sorts of reasons, including media handling, integration with other services, and network convenience. Consider then, using a zpool-in-a-file as the file format, rather than zfs send streams. Make a backup pool out of N conveniently-sized files, sparse to start with if it suits you. Go ahead and make it a raidz[23] pool, in case some of your backup media goes bad. zfs recv your backups into this pool. zfs export the pool, to quiesce the file contents. If the files are themselves in zfs, snapshot that filesystem now for good measure; then you can immediately bring your backup pool online again, as well as have local reference copies of old backups, as storage space allows. Send these files to whatever backup system/service you want to use. Handle them like any other large archive file format. If you choose file sizes well, you can burn them to dvd's or write them to tapes. Multiple smaller files (rather than one file per medium) can be easier to handle, but its best to make multiple raidz vdevs and arrange a file from each vdev per medium (so you can still recover from a lost dvd/tape). If you have a non-zfs remote storage server, rsync works well to update the remote copies incrementally after each backup cycle (and to bring back older versions later if needed). Lots of cloud backup providers exist now that can do similar incremental replication. gpg sign and encrypt the files before sending, if you need to. Some day soon, zfs crypto will allow backups encrypted within the pool, without defeating incremental replication of the files (as gpg will). Another option is to mount these files directly on whatever generic NAS devices you want to hold the backups, and import the pool from there. I'd be wary, but if you were to consider that, fortunately there's a good testing tool (zfs scrub) to help you be sure the NAS service is reliable. You do need all (or at least most) of your files available in order to do a restore. Given that you should be preferring to use local snapshots for small recovery jobs anyway, that's not really a burden for a full recovery. At this point, you get back all the snapshots that were in the backup pool at the time it was saved. The zfs send stream format used to not be committed, though it appears this has recently changed. It still has all the other drawbacks previously noted. The zpool format is committed and does have a tested and supported backwards compatibility/upgrade path. -- Dan. pgpmu7bvoOxnf.pgp Description: PGP signature ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
What dou you use instead? * * **tar cvf - /some/dir | (cd /some/other/dir; tar xf -) BTW: I recommend star and to use either the H=exustar or -dump option. Jörg i will have to check it out. I recently migrated to opensolaris from FreeBSD and i have a LOT to learn. I am really enjoying Opensolaris so far. so star is better than tar? ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. This would be annoying for the reasons mentioned - some risk of being able to restore in future (although that's a pretty small risk) and inability to restore with any granularity, i.e. you have to restore the whole FS if you restore anything at all. A better approach would be zfs send and pipe directly to zfs receive on the external media. This way, in the future, anything which can read ZFS can read the backup media, and you have granularity to restore either the whole FS, or individual things inside there. Plus, the only way to guarantee the integrity of a zfs send data stream is to perform a zfs receive on that data stream. So by performing a successful receive, you've guaranteed the datastream is not corrupt. Yet. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
Edward Ned Harvey sola...@nedharvey.com wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. NO, zfs send is not a backup. From a backup, you could restore individual files. Jörg -- EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin j...@cs.tu-berlin.de(uni) joerg.schill...@fokus.fraunhofer.de (work) Blog: http://schily.blogspot.com/ URL: http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
NO, zfs send is not a backup. From a backup, you could restore individual files. Jörg I disagree. It is a backup. It's just not an enterprise backup solution ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. This would be annoying for the reasons mentioned - some risk of being able to restore in future (although that's a pretty small risk) and inability to restore with any granularity, i.e. you have to restore the whole FS if you restore anything at all. A better approach would be zfs send and pipe directly to zfs receive on the external media. This way, in the future, anything which can read ZFS can read the backup media, and you have granularity to restore either the whole FS, or individual things inside there. There have also been comments about the extreme fragility of the data stream compared to other archive formats. In general it is strongly discouraged for these purposes. --Toby Plus, the only way to guarantee the integrity of a zfs send data stream is to perform a zfs receive on that data stream. So by performing a successful receive, you've guaranteed the datastream is not corrupt. Yet. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain t...@telegraphics.com.au wrote: On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. This would be annoying for the reasons mentioned - some risk of being able to restore in future (although that's a pretty small risk) and inability to restore with any granularity, i.e. you have to restore the whole FS if you restore anything at all. A better approach would be zfs send and pipe directly to zfs receive on the external media. This way, in the future, anything which can read ZFS can read the backup media, and you have granularity to restore either the whole FS, or individual things inside there. There have also been comments about the extreme fragility of the data stream compared to other archive formats. In general it is strongly discouraged for these purposes. Yet it is used in ZFS flash archives on Solaris 10 and are slated for use in the successor to flash archives. This initial proposal seems to imply using the same mechanism for a system image backup (instead of just system provisioning). http://mail.opensolaris.org/pipermail/caiman-discuss/2010-January/015909.html -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zfs send/receive as backup - reliability?
On 16-Jan-10, at 6:51 PM, Mike Gerdts wrote: On Sat, Jan 16, 2010 at 5:31 PM, Toby Thain t...@telegraphics.com.au wrote: On 16-Jan-10, at 7:30 AM, Edward Ned Harvey wrote: I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. You don't need to store the zfs send data stream on your backup media. This would be annoying for the reasons mentioned - some risk of being able to restore in future (although that's a pretty small risk) and inability to restore with any granularity, i.e. you have to restore the whole FS if you restore anything at all. A better approach would be zfs send and pipe directly to zfs receive on the external media. This way, in the future, anything which can read ZFS can read the backup media, and you have granularity to restore either the whole FS, or individual things inside there. There have also been comments about the extreme fragility of the data stream compared to other archive formats. In general it is strongly discouraged for these purposes. Yet it is used in ZFS flash archives on Solaris 10 I can see the temptation, but isn't it a bit under-designed? I think Mr Nordin might have ranted about this in the past... --Toby and are slated for use in the successor to flash archives. This initial proposal seems to imply using the same mechanism for a system image backup (instead of just system provisioning). http://mail.opensolaris.org/pipermail/caiman-discuss/2010-January/ 015909.html -- Mike Gerdts http://mgerdts.blogspot.com/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zfs send/receive as backup - reliability?
Hi, I am considering building a modest sized storage system with zfs. Some of the data on this is quite valuable, some small subset to be backed up forever, and I am evaluating back-up options with that in mind. My understanding is that zfs send approximately captures the copy-on-write file system block-level dump, and zfs receive plays it back to rebuild the file system, and this can be used among other things for back-ups. I call the dump stream below. How reliable is this? I don't mind the fact I would have to replay entire file system instead of individual files. My concern is that for whatever reason I'd lose ability to play the stream back, and would not be able to restore possibly years from now. Is the format documented? Is it possible to interpret the data with independent tools, like it's possible with tar/pax/cpio archives? Even if no such tool exists now, could I for example write a user space tool using currently existing open source zfs user space library that was able to extract useful information from the data stream? I realise this could be somewhat complex, especially incremental dumps - but just how hard? How exactly does the stream have to match the file system to restore? My assumption zfs requires an exact match: you can only restore at exact point you backed up from. (Fine by me, just need to know.) Follow-up: does one corrupted incremental back-up set invalidate all incremental back-up sets until the next full back-up point (or higher-level incremental point)? Assuming the zfs send data stream hasn't been corrupted, have there been instances where it's not possible to restore file system by playing it back via zfs receive? Have there been cases where some bug has caused zfs send data become corrupted so that restoration is no longer possible? (Either zfs on-disk file system bug, or something in zfs send not doing the right thing.) Is it possible to make dry-run restoration attempt at back-up time to verify the restoration would succeed? Would you recommend zfs send/receive as a back-up strategy for highly valuable data at this point in time? Ignoring the individual file vs. whole file system aspect, how reliable you rank it compared to tar/pax? Regards, Lassi ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
