subject:"\[zfs\-discuss\] zfs send\/receive as backup \- reliability\?"

Ragnar Sundblad  wrote:

> Yes! Modern LTO drives can typically vary their speed about a factor four
> or so, so even if you can't keep up with the tape drive maximum speed,
> it will typically work pretty good anyway. If you can't keep up even then,
> it will have to stop, back up a bit, and restart, which will be _very_
> slow. Having a disk system deliver data at 240 MB/s at the same time
> as you are writing to it can be a bit of a challenge. 

And star implements a FIFO that is written in a way that dramatically reduces 
the sawtooth behavior seen typically with other applications. You just need to 
tell star to use a say 2 GB FIFO and star will be able to keep the tame 
streaming for a longer time before it waits until there is enough data for the 
next longer streaming period.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad

On 21 jan 2010, at 00.20, Al Hopper wrote:

> I remember for about 5 years ago (before LT0-4 days) that streaming
> tape drives would go to great lengths to ensure that the drive kept
> streaming - because it took so much time to stop, backup and stream
> again.  And one way the drive firmware accomplished that was to write
> blocks of zeros when there was no data available.  This also occurred
> when the backup source was sending a bunch of small files, which took
> longer to stream and did'nt produce enough data to keep the drive
> writing useful data.  And if you had the tape hardware setup to do
> compression, then, assuming a normal 2:1 compression ratio, you'd need
> to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec.  The
> net result was the consumption of a lot more tape than a
> back-of-the-napkin calculation told you was required.
> 
> Obviously at higher compression ratios or with the higher stream data
> write rates you quote below - this problem becomes more troublesome.
> So I agree with your conclusion: "The only way to realistically feed
> that is from disk."

Yes! Modern LTO drives can typically vary their speed about a factor four
or so, so even if you can't keep up with the tape drive maximum speed,
it will typically work pretty good anyway. If you can't keep up even then,
it will have to stop, back up a bit, and restart, which will be _very_
slow. Having a disk system deliver data at 240 MB/s at the same time
as you are writing to it can be a bit of a challenge. 

I haven't seen drives that fill out with zeros. Sounds like an ugly
solution, but maybe it could be useful in some strange case.

/ragge s

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad


On 20 jan 2010, at 17.22, Julian Regel wrote:

> >It is actually not that easy.
> >
> >Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
> >
> >Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare 
> >+ 2x OS disks.
> >The four raidz2 group form a single pool. This would provide well over 
> >30TB of logical storage per each box.
> >
> >Now you rsync all the data from your clients to a dedicated filesystem 
> >per client, then create a snapshot.
> >All snapshots are replicated to a 2nd x4540 so even if you would loose 
> >entire box/data for some reason you would still have a spare copy.
> >
> >Now compare it to a cost of a library, lto drives, tapes, software + 
> >licenses, support costs, ...
> >
> >See more details at 
> >http://milek.blogspot.com/2009/12/my-presentation-at-losug.html
> 
> I've just read your presentation Robert. Interesting stuff.
> 
> I've also just done a pen and paper exercise to see how much 30TB of tape 
> would cost as a comparison to your disk based solution.
> 
> Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x 
> LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped 
> SL48 despite the Sun website saying it's a supported option. Each LTO3 has a 
> native capacity of 300GB and the SL48 can hold up to 48 tapes in the library 
> (14.4TB native per library). To match the 30TB in your solution, we'd need 
> two libraries totalling £28000.

LTO3 has native capacity 400 GB, LTO4 has 800. The price is about the same per 
tape and per drive, a little higher for LTO4.

> You would also need 100 LTO3 tapes to provide 30TB of native storage. I 
> recently bought a pack of 20 tapes for £340, so five packs would be £1700.

Or rather 37 LTO4 tapes, and only one 48 tape library. But that doesn't matter, 
the interesting part is that one now can use whatever best solves the problem 
at hand.

> So you could provision a tape backup for just under £3 (~$49000). In 
> comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price 
> £30900. I've not factored in backup software since you could use an open 
> source solution such as Amanda or Bacula.
> 
> Which isn't to say tape would be a "better" solution since it's going to be 
> slower to restore etc. But it does show that tape can work out cheaper, 
> especially since the cost of a high speed WAN link isn't required.

Reading from tape is normally faster than reading from (a single) disk. Seek 
time of course isn't.

/ragge

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Al Hopper

On Wed, Jan 20, 2010 at 2:52 PM, David Magda  wrote:
>
> On Jan 20, 2010, at 12:21, Robert Milkowski wrote:
>
>> On 20/01/2010 16:22, Julian Regel wrote:
>>>
>> [...]
>>>
>>> So you could provision a tape backup for just under £3 (~$49000). In
>>> comparison, the cost of one X4540 with ~ 36TB usable storage is UK list
>>> price £30900. I've not factored in backup software since you could use an
>>> open source solution such as Amanda or Bacula.
>>
>> [...]
>> You would also need to add at least one server to your library with fc
>> cards.
>> Then with most software you would need more tapes due to data
>> fragmentation and a need to do regular full backups (with zfs+rsync you only
>> do a full backup once).
>>
>> So in best case a library will cost about the same as disk based solution
>> but generally will be less flexible, etc. If you would add any enterprise
>> software on top of it (Legato, NetBackup, ...) then the price would change
>> dramaticallly. Additionally with ZFS one could start using deduplication (in
>> testing already).
>
> Regardless of the economics of tape, nowadays you generally need to go to
> disk first because trying to stream at 120 MB/s (LTO-4) really isn't
> practical over the network, directly from the client.

I remember for about 5 years ago (before LT0-4 days) that streaming
tape drives would go to great lengths to ensure that the drive kept
streaming - because it took so much time to stop, backup and stream
again.  And one way the drive firmware accomplished that was to write
blocks of zeros when there was no data available.  This also occurred
when the backup source was sending a bunch of small files, which took
longer to stream and did'nt produce enough data to keep the drive
writing useful data.  And if you had the tape hardware setup to do
compression, then, assuming a normal 2:1 compression ratio, you'd need
to source 240Mb/Sec in order to keep the tape writing 120Mb/Sec.  The
net result was the consumption of a lot more tape than a
back-of-the-napkin calculation told you was required.

Obviously at higher compression ratios or with the higher stream data
write rates you quote below - this problem becomes more troublesome.
So I agree with your conclusion: "The only way to realistically feed
that is from disk."

> So in the end you'll be starting with disk (either DAS or VTL or whatever),
> and generally going to tape if you need to keep stuff that's older than
> (say) 3-6 months. Tape also doesn't rotate while it's sitting there, so if
> it's going to be sitting around for a while (e.g., seven years) better to
> use tape than something that sucks up power.
>
> LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB and
> (uncompressed) writes at 180 MB/s. The only way to realistically feed that
> is from disk.
>
> ___

Regards,

-- 
Al Hopper  Logical Approach Inc,Plano,TX a...@logical-approach.com
   Voice: 972.379.2133 Timezone: US CDT
OpenSolaris Governing Board (OGB) Member - Apr 2005 to Mar 2007
http://www.opensolaris.org/os/community/ogb/ogb_2005-2007/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Miles Nordin  wrote:

> From the perspective of MY business, I would much rather have the dark
> OOB acl/fork/whatever-magic that's gone into ZFS and NFSv4 supported
> in standard tools like rsync and GNUtar.  This is, for example, what

GNU tar does not support any platform speficic feature on any OS.
Don't expect that GNU tar will ever add such properties..

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Ian Collins  wrote:

> > We are talking about TAR and I did give a pointer to the star archive 
> > format 
> > documentation, so it is obvious that I was talking about the ACL format from
> > Sun tar. This format is not documented.
> >
> >   
> It is, Sun's ZFS ACL aware tools use acltotext() to format ACLs.

Please don't reply without checking facts.

The fact that you know that there is salt in the soup does not give
you the whole list of ingredients Please look into the Sun tar format
to understand that you are wrong.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Magda



On Jan 20, 2010, at 12:21, Robert Milkowski wrote:


On 20/01/2010 16:22, Julian Regel wrote:



[...]
So you could provision a tape backup for just under £3 (~ 
$49000). In comparison, the cost of one X4540 with ~ 36TB usable  
storage is UK list price £30900. I've not factored in backup  
software since you could use an open source solution such as Amanda  
or Bacula.

[...]
You would also need to add at least one server to your library with  
fc cards.
Then with most software you would need more tapes due to data  
fragmentation and a need to do regular full backups (with zfs+rsync  
you only do a full backup once).


So in best case a library will cost about the same as disk based  
solution but generally will be less flexible, etc. If you would add  
any enterprise software on top of it (Legato, NetBackup, ...) then  
the price would change dramaticallly. Additionally with ZFS one  
could start using deduplication (in testing already).


Regardless of the economics of tape, nowadays you generally need to go  
to disk first because trying to stream at 120 MB/s (LTO-4) really  
isn't practical over the network, directly from the client.


So in the end you'll be starting with disk (either DAS or VTL or  
whatever), and generally going to tape if you need to keep stuff  
that's older than (say) 3-6 months. Tape also doesn't rotate while  
it's sitting there, so if it's going to be sitting around for a while  
(e.g., seven years) better to use tape than something that sucks up  
power.


LTO-5 is expected to be released RSN, with a native capacity of 1.6 TB  
and (uncompressed) writes at 180 MB/s. The only way to realistically  
feed that is from disk.


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Miles Nordin

> "jr" == Julian Regel  writes:

jr> While I am sure that star is technically a fine utility, the
jr> problem is that it is effectively an unsupported product.

I have no problems with this whatsoever.

jr> If our customers find a bug in their backup that is caused by
jr> a failure in a Sun supplied utility, then they have a legal
jr> course of action. The customer's system administrators are
jr> covered because they were using tools provided by the
jr> vendor. The wrath of the customer would be upon Sun, not the
jr> supplier (us) or the supplier's technical lead (me).

We were just talking about this somewhere else, actually: ``if
something goes wrong, its their ass. but if nothing ever gets done,
its nobody's fault.''  It's sad for me how much money is to be made
supporting broken corporate cultures like that.

I'm not saying you're wrong, just that you might not want to
contribute to such a culture because you've chosen to endure it for a
scratch.  You need to have a better way to evaluate employees than
micromanagement-by-the-clueless and vindictive hindsight.  But the
point that there's money to be made by bleeding it out of ossified
broken American companies is well-taken.

jr> From the perspective of the business, the system administrator
jr> will have acted irresponsibly by choosing a tool that has no
jr> vendor support.

From the perspective of MY business, I would much rather have the dark
OOB acl/fork/whatever-magic that's gone into ZFS and NFSv4 supported
in standard tools like rsync and GNUtar.  This is, for example, what
Apple achieved with CUPS and why I can share printers between Ubuntu
and Mac OS effortlessly, and this increases the amount of money I'm
willing to give Apple for their proprietary platform.  The purpose of
the tool I'm discussing definitely includes the same level of
cooperation, so working with the existing best-in-class and
most-popular tools, and reasonableness, might be better than brittle
CYA support in some fringey '/opt/SUNWbkpkit/bin/VendorCP -Rf' tool.

Even if you get your cyaCP tool you may find it doesn't achieve the
ass-covering you wanted because these tools can be cheeky little
bastards.  Most of the other quirky little balkanized-platform
Solaris-only tools are littered with straightjacketing assertions to
avoid ``call generators'' and push the blame back onto the sysadmin,
then there is some ``all bets are off'' flag to allowe you to actually
accomplish job, like 'NOINUSECHECK=1 format -e'.  

Honestly...why bother playing this game?


pgpBTS02xCBZW.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins


Joerg Schilling wrote:

Ian Collins  wrote:

  

The correct way to archivbe ACLs would be to put them into extended POSIX tar
attrubutes as star does.

See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.

ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz

The ACL format used by Sun is undocumented..

  
  

man acltotext



We are talking about TAR and I did give a pointer to the star archive format 
documentation, so it is obvious that I was talking about the ACL format from

Sun tar. This format is not documented.

  

It is, Sun's ZFS ACL aware tools use acltotext() to format ACLs.

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins

Julian Regel wrote:

>It is actually not that easy.
>
>Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
>
>Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare
>+ 2x OS disks.
>The four raidz2 group form a single pool. This would provide well over
>30TB of logical storage per each box.
>
>Now you rsync all the data from your clients to a dedicated filesystem
>per client, then create a snapshot.
>All snapshots are replicated to a 2nd x4540 so even if you would loose
>entire box/data for some reason you would still have a spare copy.
>
>Now compare it to a cost of a library, lto drives, tapes, software +
>licenses, support costs, ...
>
>See more details at
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

I've just read your presentation Robert. Interesting stuff.

I've also just done a pen and paper exercise to see how much 30TB of 
tape would cost as a comparison to your disk based solution.

Using list prices from Sun's website (and who pays list..?), an SL48 
with 2 x LTO3 drives would cost £14000. I couldn't see a price on an 
LTO4 equipped SL48 despite the Sun website saying it's a supported 
option. Each LTO3 has a native capacity of 300GB and the SL48 can hold 
up to 48 tapes in the library (14.4TB native per library). To match 
the 30TB in your solution, we'd need two libraries totalling £28000.

You would also need 100 LTO3 tapes to provide 30TB of native storage. 
I recently bought a pack of 20 tapes for £340, so five packs would be 
£1700.

So you could provision a tape backup for just under £3 (~$49000). 
In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
list price £30900. I've not factored in backup software since you 
could use an open source solution such as Amanda or Bacula.

A more apples to apples comparison would be to compare the storage 
only.  Both removable drive and tape options require a server with FC or 
SCSI ports, so that can be excluded from the comparison.

So for 30TB, assuming 2TB drives @ ~£100 with a pool built of 6 drive 
raidz vdevs 18 drives would be required plus 2 16 drive shelves.  So 
each backup set would cost about £1800.  So there's not a great deal of 
difference.  With drives you also get the added benefit of keeping all 
your incrementals (as snapshots) on the archive set.

HDD price per GB will continue to drop faster than tape, so it will be 
interesting to do the same comparison in 12 months.

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Richard Elling

On Jan 20, 2010, at 3:15 AM, Joerg Schilling wrote:

> Richard Elling  wrote:
> 
>>> 
>>> ufsdump/restore was perfect in that regard.  The lack of equivalent 
>>> functionality is a big problem for the situations where this functionality 
>>> is a business requirement.
>> 
>> How quickly we forget ufsdump's limitations :-).  For example, it is not 
>> supported
>> for use on an active file system (known data corruption possibility) and 
>> UFS snapshots are, well, a poor hack and often not usable for backups.
>> As the ufsdump(1m) manpage says,
> 
> It seems you forgot that zfs also needs snapshots. There is nothing bad with 
> snapshots.

Yes, snapshots are a good thing. But most people who try fssnap 
on the UFS root file system will discover that it doesn't work; for 
reasons mentioned in the NOTES section of fssnap_ufs(1m). 
fssnap_ufs is simply a butt-ugly hack. So if you believe you can
reliably use ufsdump to store a DR copy of root for a 7x24x365 
production environment, then you probably believe the Backup
Fairy will leave a coin under your pillow when your restore fails :-)

Fortunately, ZFS snapshot do the right thing.
 -- richard

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

On 20/01/2010 17:21, Robert Milkowski wrote:

On 20/01/2010 16:22, Julian Regel wrote:

>It is actually not that easy.
>
>Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
>
>Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot 
spare

>+ 2x OS disks.
>The four raidz2 group form a single pool. This would provide well over
>30TB of logical storage per each box.
>
>Now you rsync all the data from your clients to a dedicated filesystem
>per client, then create a snapshot.
>All snapshots are replicated to a 2nd x4540 so even if you would loose
>entire box/data for some reason you would still have a spare copy.
>
>Now compare it to a cost of a library, lto drives, tapes, software +
>licenses, support costs, ...
>
>See more details at
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

I've just read your presentation Robert. Interesting stuff.

I've also just done a pen and paper exercise to see how much 30TB of 
tape would cost as a comparison to your disk based solution.

Using list prices from Sun's website (and who pays list..?), an SL48 
with 2 x LTO3 drives would cost £14000. I couldn't see a price on an 
LTO4 equipped SL48 despite the Sun website saying it's a supported 
option. Each LTO3 has a native capacity of 300GB and the SL48 can 
hold up to 48 tapes in the library (14.4TB native per library). To 
match the 30TB in your solution, we'd need two libraries totalling 
£28000.

You would also need 100 LTO3 tapes to provide 30TB of native storage. 
I recently bought a pack of 20 tapes for £340, so five packs would be 
£1700.

So you could provision a tape backup for just under £3 (~$49000). 
In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
list price £30900. I've not factored in backup software since you 
could use an open source solution such as Amanda or Bacula.

Which isn't to say tape would be a "better" solution since it's going 
to be slower to restore etc. But it does show that tape can work out 
cheaper, especially since the cost of a high speed WAN link isn't 
required.

JR

You would also need to add at least one server to your library with fc 
cards.
Then with most software you would need more tapes due to data 
fragmentation and a need to do regular full backups (with zfs+rsync 
you only do a full backup once).

So in best case a library will cost about the same as disk based 
solution but generally will be less flexible, etc. If you would add 
any enterprise software on top of it (Legato, NetBackup, ...) then the 
price would change dramaticallly. Additionally with ZFS one could 
start using deduplication (in testing already).

What I really mean is that a disk based solution used to be much more 
expensive than tapes but currently they are comparable in costs while 
often the disk based is more flexible.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

On 20/01/2010 16:22, Julian Regel wrote:

>It is actually not that easy.
>
>Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
>
>Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare
>+ 2x OS disks.
>The four raidz2 group form a single pool. This would provide well over
>30TB of logical storage per each box.
>
>Now you rsync all the data from your clients to a dedicated filesystem
>per client, then create a snapshot.
>All snapshots are replicated to a 2nd x4540 so even if you would loose
>entire box/data for some reason you would still have a spare copy.
>
>Now compare it to a cost of a library, lto drives, tapes, software +
>licenses, support costs, ...
>
>See more details at
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

I've just read your presentation Robert. Interesting stuff.

I've also just done a pen and paper exercise to see how much 30TB of 
tape would cost as a comparison to your disk based solution.

Using list prices from Sun's website (and who pays list..?), an SL48 
with 2 x LTO3 drives would cost £14000. I couldn't see a price on an 
LTO4 equipped SL48 despite the Sun website saying it's a supported 
option. Each LTO3 has a native capacity of 300GB and the SL48 can hold 
up to 48 tapes in the library (14.4TB native per library). To match 
the 30TB in your solution, we'd need two libraries totalling £28000.

You would also need 100 LTO3 tapes to provide 30TB of native storage. 
I recently bought a pack of 20 tapes for £340, so five packs would be 
£1700.

So you could provision a tape backup for just under £3 (~$49000). 
In comparison, the cost of one X4540 with ~ 36TB usable storage is UK 
list price £30900. I've not factored in backup software since you 
could use an open source solution such as Amanda or Bacula.

Which isn't to say tape would be a "better" solution since it's going 
to be slower to restore etc. But it does show that tape can work out 
cheaper, especially since the cost of a high speed WAN link isn't 
required.

JR

You would also need to add at least one server to your library with fc 
cards.
Then with most software you would need more tapes due to data 
fragmentation and a need to do regular full backups (with zfs+rsync you 
only do a full backup once).

So in best case a library will cost about the same as disk based 
solution but generally will be less flexible, etc. If you would add any 
enterprise software on top of it (Legato, NetBackup, ...) then the price 
would change dramaticallly. Additionally with ZFS one could start using 
deduplication (in testing already).

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Miles Nordin

> "ae" == Allen Eastwood  writes:
> "ic" == Ian Collins  writes:

 >> If people are really still backing up to tapes or DVD's, just
 >> use file vdev's, export the pool, and then copy the unmounted
 >> vdev onto the tape or DVD.

ae> And some of those enterprises require backup mechanism that
ae> can be easily used in a DR situation.

ae> ufsdump/restore was perfect in that regard.  The lack of
ae> equivalent functionality is a big problem for the situations
ae> where this functionality is a business requirement.

ae> For example, one customer, local government, requires a backup
ae> that can be taken offsite and used in a DR situation.

Were you confused by some part of:

 Use file vdevs, epxort the pool, and then copy the unmounted vdev
 onto the tape.

or do you find that this doesn't do what you want?  because it seems
fine to me.  And the fact that it doesn't need any extra tools means
it's unlikely to break (1) far into the future or (2) for a few
unlucky builds, and (3) that the restore environment is simple and
doesn't involve prepopulating some Legato Database with the TOC of
every tape in the library or some such nonsense, which ought to all be
among your ``requirements'' but if you're substituting for those,
``works in exactly the way we were used to it working before'' then
you may as well use 'zfs send' since you're more concerned with
identical-feeling invocatoin syntax than the problems I mentioned.

ic> For a full recovery, you can archive a send stream and receive
ic> it back.

You can send the stream to the tape, transport the tape to the DR
site, and receive it.  You can do this weekly as part of your offsite
backup plan provided that you receive each tape you transport
immediately.  Then the data should be permanently stored on disk at
the DR site, and the tapes used only for transport.

If you store the backup permanently on tape then it's a step backwards
from tar/cpio/ufsrestore because the 'zfs send' format is more fragile
and has to be restored entire.  If you receive the tape immediately
this is an improvement because under the old convention tapes could be
damaged in transit, or over the years by heat/dust/sunlight, without
your knowledge, while on disks it's simple to scrub periodically.

I am not trying to take away your tapes, Allen, so please quote Ian
instead if that's the thing you object to.  I've instead suggested a
different way to use them if you really do need them archivally: store
file vdev's on them.  If you're just using them to replicate data to
the DR site then you needn't even go as far as my workaround.

I do agree that there's a missing tool: it's not possible to copy one
subdirectory to another while preserving holes, forkey extended
attributes, and ACL's.  Also if Windows ACL's are going to be stored
right in the filesystem, then Windows ACL's probably ought to be
preserved over an rsync pipe between Solaris and EnTee, or a
futuristic tarball written on one and extracted on the other.  I don't
agree that the missing tool is designed primarily for the narrow
use-case of writing to ancient backup tapes: it's a more general tool.
or, really, it's just a matter of documenting and committing the
extra-OOB-gunk APIs and then fixing rsync and GNUtar.

pgpk5wk2koJcZ.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

>It is actually not that easy.
>
>Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.
>
>Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare 
>+ 2x OS disks.
>The four raidz2 group form a single pool. This would provide well over 
>30TB of logical storage per each box.
>
>Now you rsync all the data from your clients to a dedicated filesystem 
>per client, then create a snapshot.
>All snapshots are replicated to a 2nd x4540 so even if you would loose 
>entire box/data for some reason you would still have a spare copy.
>
>Now compare it to a cost of a library, lto drives, tapes, software + 
>licenses, support costs, ...
>
>See more details at 
>http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

I've just read your presentation Robert. Interesting stuff.

I've also just done a pen and paper exercise to see how much 30TB of tape would 
cost as a comparison to your disk based solution.

Using list prices from Sun's website (and who pays list..?), an SL48 with 2 x 
LTO3 drives would cost £14000. I couldn't see a price on an LTO4 equipped SL48 
despite the Sun website saying it's a supported option. Each LTO3 has a native 
capacity of 300GB and the SL48 can hold up to 48 tapes in the library (14.4TB 
native per library). To match the 30TB in your solution, we'd need two 
libraries totalling £28000.

You would also need 100 LTO3 tapes to provide 30TB of native storage. I 
recently bought a pack of 20 tapes for £340, so five packs would be £1700.


So you could provision a tape backup for just under £3 (~$49000). In 
comparison, the cost of one X4540 with ~ 36TB usable storage is UK list price 
£30900. I've not factored in backup software since you could use an open source 
solution such as Amanda or Bacula.

Which isn't to say tape would be a "better" solution since it's going to be 
slower to restore etc. But it does show that tape can work out cheaper, 
especially since the cost of a high speed WAN link isn't required.

JR



  ___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Bob Friesenhahn


On Wed, 20 Jan 2010, Julian Regel wrote:


If our customers find a bug in their backup that is caused by a 
failure in a Sun supplied utility, then they have a legal course of 
action. The customer's system administrators are covered because 
they were using tools provided by the vendor. The wrath of the 
customer would be upon Sun, not the supplier (us) or the supplier's 
technical lead (me).


I would love to try whatever you are smoking because it must be really 
good stuff.  It would be a bold new step for me, but the benefits are 
clear.


While your notions of the transitive protection offered by vendor 
"support" are interesting, I will be glad to meet you in the 
unemployment line then we can share some coffee and discuss the good 
old days.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Dyer-Bennet

On Wed, January 20, 2010 04:48, Ragnar Sundblad wrote:

> LTO media is still cheaper than equivalent sized disks, maybe a factor 5
> or so. LTO drivers cost a little, but so do disk shelves. So, now that
> there is no big price issue, there is choice instead. Use it!

Depends on the scale you're operating at.

Backing up my 800GB home data pool onto a couple of external 1TB USB
drives is *immensely* cheaper than buying tape equipment.

At enough bigger scales, I accept that tape is still cheaper.  Makes
sense, since the tapes are relatively simple compared to drives, and you
only need a small number of drives to use a large number of tapes.

I think hard drives are still cheaper at small-enterprise levels, actually.

-- 
David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread David Dyer-Bennet

On Wed, January 20, 2010 09:23, Robert Milkowski wrote:

> Now you rsync all the data from your clients to a dedicated filesystem
> per client, then create a snapshot.

Is there an rsync out there that can reliably replicate all file
characteristics between two ZFS/Solaris systems?  I haven't found one. 
The ZFS ACLs seem to be beyond all of them, in particular.

(Losing just that, and preserving the data, is clearly far, far better
than losing everything!  And a system build *knowing* it was losing the
protections could preserve them some other way.)

-- 
David Dyer-Bennet, d...@dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


On 20/01/2010 11:26, Joerg Schilling wrote:

Edward Ned Harvey  wrote:

   

Star implements this in a very effective way (by using libfind) that is
even
faster that the find(1) implementation from Sun.
   

Even if I just "find" my filesystem, it will run for 7 hours.  But zfs can
create my whole incremental snapshot in a minute or two.  There is no way
star or any other user-space utility that walks the filesystem can come
remotely close to this performance.  Such performance can only be
implemented at the filesystem level, or lower.
 

You claim that it is fast for you but this is because it is block oriented and
because you probably changed only few data.

If you like to have a backup that allows to access files, you need a file based
backup and I am sure that even a filesystem level scan for recently changed
files will not be much faster than what you may achive with e.g. star.

   


Not really - I mean if you do an incremental zfs send and on the other 
side receive it so a filesystem/snapshot is created then you have an 
access to each individual file while having the benefit of doing block 
level incremental zfs send which in many environments will be way 
faster then rsync/start/...


--
Robert Milkowski
http://milek.blogspot.com


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


On 20/01/2010 10:48, Ragnar Sundblad wrote:

On 19 jan 2010, at 20.11, Ian Collins wrote:

   

Julian Regel wrote:
 

Based on what I've seen in other comments, you might be right. Unfortunately, I 
don't feel comfortable backing up ZFS filesystems because the tools aren't 
there to do it (built into the operating system or using Zmanda/Amanda).

   

Commercial backup solutions are available for ZFS.
 

I know tape backup isn't sexy, but it's a reality for many of us and it's not 
going away anytime soon.

   

True, but I wonder how viable its future is.  One of my clients requires 17 
LT04 types for a full backup, which cost more and takes up more space than the 
equivalent in removable hard drives.

In the past few years growth in hard drive capacities has outstripped tapes to 
the extent that removable hard drives and ZFS snapshots have become a more cost 
effective and convenient backup media.
 

LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. 
LTO drivers cost a little, but so do disk shelves. So, now that there is no big 
price issue, there is choice instead. Use it!

Hard drives are good for random access - both restore of individual files and 
partial rewrite.

Hard drivers aren't faster than tape for data transfer, but they might be 
cheaper to run in parallel and therefore you could potentially gain speed. Hard 
drives have shorter seek time, which may be important.

Hard drives are probably bad for storing for longer times - especially - you 
will never know how long it could be stored before it will fail. A month? 
Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes 
are supposed to be able to keep it's data for at least 30 years of stored 
properly. Hard drives are probably best when used online or at least very often.

So - it is wrong to say that one is better or cheaper than the other. They have 
different properties, and could be used to solve different problems.


   


It is actually not that easy.

Compare a cost of 2x x4540 with 1TB disks to equivalent solution on LTO.

Each x4540 could be configured as: 4x 11 disks in raidz-2 + 2x hot spare 
+ 2x OS disks.
The four raidz2 group form a single pool. This would provide well over 
30TB of logical storage per each box.


Now you rsync all the data from your clients to a dedicated filesystem 
per client, then create a snapshot.
All snapshots are replicated to a 2nd x4540 so even if you would loose 
entire box/data for some reason you would still have a spare copy.


Now compare it to a cost of a library, lto drives, tapes, software + 
licenses, support costs, ...


See more details at 
http://milek.blogspot.com/2009/12/my-presentation-at-losug.html


--
Robert Milkowski
http://milek.blogspot.com

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


On 19/01/2010 19:11, Ian Collins wrote:

Julian Regel wrote:


Based on what I've seen in other comments, you might be right. 
Unfortunately, I don't feel comfortable backing up ZFS filesystems 
because the tools aren't there to do it (built into the operating 
system or using Zmanda/Amanda).



Commercial backup solutions are available for ZFS.
I know tape backup isn't sexy, but it's a reality for many of us and 
it's not going away anytime soon.


True, but I wonder how viable its future is.  One of my clients 
requires 17 LT04 types for a full backup, which cost more and takes up 
more space than the equivalent in removable hard drives.


In the past few years growth in hard drive capacities has outstripped 
tapes to the extent that removable hard drives and ZFS snapshots have 
become a more cost effective and convenient backup media.


What do people with many tens of TB use for backup these days?


http://milek.blogspot.com/2009/12/my-presentation-at-losug.html

--
Robert Milkowski
http://milek.blogspot.com

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Michael Schuster


Joerg Schilling wrote:

Julian Regel  wrote:

If you like to have a backup that allows to access files, you need a file based 
backup and I am sure that even a filesystem level scan for recently changed 
files will not be much faster than what you may achive with e.g. star.


Note that ufsdump directly accesees the raw disk device and thus _is_ at 
filesystem leven but still is slower than star on UFS.

While I am sure that star is technically a fine utility, the problem is that it 
is effectively an unsupported product.


From this viewpoint, you may call most of Solaris "unsupported".


what is that supposed to mean?

Michael
--
Michael Schusterhttp://blogs.sun.com/recursion
Recursion, n.: see 'Recursion'
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Julian Regel  wrote:

> >> While I am sure that star is technically a fine utility, the problem is 
> >> that it is effectively an unsupported product.
>
> >From this viewpoint, you may call most of Solaris "unsupported".
>
> From the perspective of the business, the contract with Sun provides that 
> support.

>From a perspective of reality, such a contract will not help.

> >Do you really believe that Sun will help such a customer?
> >There are many bugs in Solaris (I remember e.g. some showstopper
> >bugs in the multimedia area) that are not fixed although they are known
> >since a very long time (more than a year).
> >There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even 
> >before) that is not fixed. As a result in many cases ACLs are not restored.
>
> If Sun don't fix a critical bug that is affecting the availability of
> server that is under support, then it becomes a problem for the legal
> department. In the ACL example, it's possible the effected users didn't have 
> a support contract.

What you seem to point out is that in case of a problem for a customer with a 
contract, the legal department gets involved. Unfortunately, laywers do not fix 
bugs.

> >Note that bugs in star are fixed much faster and looking back at the 28 years
> >of history with star, I know of not a single bug that took more than 3 months
> >to get a fix. Typically, bugs are fixed withing less than a week - many bugs
> >even within a few hours. This is a support quality that Sun does not offer.
>
> Possibly, but there is no guarantee that it will be fixed, no-one to call 
> when there is a problem, no-one to escalate the problem to if it is ignored, 
> and no company to sue if it all goes wrong.

Escalating a problem does not fix it. 

> >So please explain us where you see a problem with star..
>
> Hopefully my above comments explain sufficiently. It's not a technical issue 
> with star, it's a business issue. The rules there are very different and not 
> based on merit (this is also why many companies prefer running their mission 
> critical apps on Red Hat Enterprise Linux instead of CentOS, even though 
> technically they are almost identical).

Now we are back to reality.

A person that is interested in a solution will usually check what happened
in similar cases before. If you compare star with Sun supplied tools with
this background, Sun cannot outperform star.

"Red Hat Enterprise Linux" may offer something you cannot get with CentOS.
But I don't see that Sun can offer something you don't get with star.

Let me make another reality check:

Many people use GNU tar for backup purposes, but my first automated test case
with incremental backups using GNU tar did fail so miserably that I was unable
to use GNU tar as a test reference at all.

On the other side, I am doing incremental backup _and_ restore tests with 
gigabytes of real delta data on a dayly base since 2004 and I did hot see any 
problem since April 2005.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

>> While I am sure that star is technically a fine utility, the problem is that 
>> it is effectively an unsupported product.

>From this viewpoint, you may call most of Solaris "unsupported".

From the perspective of the business, the contract with Sun provides that 
support.

>> If our customers find a bug in their backup that is caused by a failure in a 
>> Sun supplied utility, then they have a legal course of action. The 
>> >>customer's system administrators are covered because they were using tools 
>> provided by the vendor. The wrath of the customer would be upon >>Sun, not 
>> the supplier (us) or the supplier's technical lead (me).

>Do you really believe that Sun will help such a customer?
>There are many bugs in Solaris (I remember e.g. some showstopper
>bugs in the multimedia area) that are not fixed although they are known
>since a very long time (more than a year).
>There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even 
>before) that is not fixed. As a result in many cases ACLs are not restored.

If Sun don't fix a critical bug that is affecting the availability of
server that is under support, then it becomes a problem for the legal
department. In the ACL example, it's possible the effected users didn't have a 
support contract.

>Note that bugs in star are fixed much faster and looking back at the 28 years
>of history with star, I know of not a single bug that took more than 3 months
>to get a fix. Typically, bugs are fixed withing less than a week - many bugs
>even within a few hours. This is a support quality that Sun does not offer.

Possibly, but there is no guarantee that it will be fixed, no-one to call when 
there is a problem, no-one to escalate the problem to if it is ignored, and no 
company to sue if it all goes wrong.

>So please explain us where you see a problem with star..

Hopefully my above comments explain sufficiently. It's not a technical issue 
with star, it's a business issue. The rules there are very different and not 
based on merit (this is also why many companies prefer running their mission 
critical apps on Red Hat Enterprise Linux instead of CentOS, even though 
technically they are almost identical).

JR



  ___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Julian Regel  wrote:

> > If you like to have a backup that allows to access files, you need a file 
> > based 
>
> > backup and I am sure that even a filesystem level scan for recently changed 
> > files will not be much faster than what you may achive with e.g. star.
> >
> > Note that ufsdump directly accesees the raw disk device and thus _is_ at 
> > filesystem leven but still is slower than star on UFS.
>
> While I am sure that star is technically a fine utility, the problem is that 
> it is effectively an unsupported product.

>From this viewpoint, you may call most of Solaris "unsupported".

> If our customers find a bug in their backup that is caused by a failure in a 
> Sun supplied utility, then they have a legal course of action. The customer's 
> system administrators are covered because they were using tools provided by 
> the vendor. The wrath of the customer would be upon Sun, not the supplier 
> (us) or the supplier's technical lead (me).

Do you really believe that Sun will help such a customer?
There are many bugs in Solaris (I remember e.g. some showstopper
bugs in the multimedia area) that are not fixed although they are known
since a very long time (more than a year).

There is a bug in ACL handling in Sun's tar (reported by me in 2004 or even 
before) that is not fixed. As a result in many cases ACLs are not restored.

Note that bugs in star are fixed much faster and looking back at the 28 years
of history with star, I know of not a single bug that took more than 3 months
to get a fix. Typically, bugs are fixed withing less than a week - many bugs
even within a few hours. This is a support quality that Sun does not offer.

So please explain us where you see a problem with star..

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

> If you like to have a backup that allows to access files, you need a file 
> based 

> backup and I am sure that even a filesystem level scan for recently changed 
> files will not be much faster than what you may achive with e.g. star.
>
> Note that ufsdump directly accesees the raw disk device and thus _is_ at 
> filesystem leven but still is slower than star on UFS.

While I am sure that star is technically a fine utility, the problem is that it 
is effectively an unsupported product.

If our customers find a bug in their backup that is caused by a failure in a 
Sun supplied utility, then they have a legal course of action. The customer's 
system administrators are covered because they were using tools provided by the 
vendor. The wrath of the customer would be upon Sun, not the supplier (us) or 
the supplier's technical lead (me).

If the system administrator has chosen star (or if the supplier recommends 
star), then the conversation becomes a lot more awkward. From the perspective 
of the business, the system administrator will have acted irresponsibly by 
choosing a tool that has no vendor support. Alternatively, the supplier will be 
held responsible for recommending a product that has broken the customer's 
ability to restore, and with no legal recourse, I wouldn't dare touch it. Sorry.

This is why Sun need to provide the solution themselves (or adopt and provide 
support for star or similar third party products).

JR



  ___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

While I can appreciate that ZFS snapshots are very useful in being able to 
recover files that users might have deleted, they do not do much to help when 
the entire disk array experiences a crash/corruption or catches fire. Backing 
up to a second array helps if a) the array is off-site and for many of us the 
cost of remote links with sufficient bandwidth is still prohibitive,or b) on 
the local network but sufficiently far away from the original array such that 
the fire does not corrupt damage the backup as well.

This leaves some form of removable storage. I'm not sure I'm aware of any 
enterprise-level removable disk solution, primarily because disk isn't really 
designed to be used for offsite backup whereas tape is.

The biggest problem with tape was finding a sufficiently large window in which 
to perform the backup. ZFS snapshots completely solves this issue, but Sun have 
failed to provide the mechanism to protect the data off-site.



  ___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Edward Ned Harvey  wrote:

> > Star implements this in a very effective way (by using libfind) that is
> > even
> > faster that the find(1) implementation from Sun.
>
> Even if I just "find" my filesystem, it will run for 7 hours.  But zfs can
> create my whole incremental snapshot in a minute or two.  There is no way
> star or any other user-space utility that walks the filesystem can come
> remotely close to this performance.  Such performance can only be
> implemented at the filesystem level, or lower.

You claim that it is fast for you but this is because it is block oriented and 
because you probably changed only few data. 

If you like to have a backup that allows to access files, you need a file based 
backup and I am sure that even a filesystem level scan for recently changed 
files will not be much faster than what you may achive with e.g. star.

Note that ufsdump directly accesees the raw disk device and thus _is_ at 
filesystem leven but still is slower than star on UFS.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Ian Collins  wrote:

> > The correct way to archivbe ACLs would be to put them into extended POSIX 
> > tar
> > attrubutes as star does.
> >
> > See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
> > documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.
> > ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz
> >
> > The ACL format used by Sun is undocumented..
> >
> >   
> man acltotext

We are talking about TAR and I did give a pointer to the star archive format 
documentation, so it is obvious that I was talking about the ACL format from
Sun tar. This format is not documented.



Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Richard Elling  wrote:

> > 
> > ufsdump/restore was perfect in that regard.  The lack of equivalent 
> > functionality is a big problem for the situations where this functionality 
> > is a business requirement.
>
> How quickly we forget ufsdump's limitations :-).  For example, it is not 
> supported
> for use on an active file system (known data corruption possibility) and 
> UFS snapshots are, well, a poor hack and often not usable for backups.
> As the ufsdump(1m) manpage says,

It seems you forgot that zfs also needs snapshots. There is nothing bad with 
snapshots.

When I was talking with Jeff Bonwick in September 2004 (before ZFS became 
public), the only feature that was missing in Solaris for a 100% correct
backup based on star was an interface for holey files, so we designed it.

I believe the only mistake from ufsdumps is that is does not use standard
OS interfaces and that it does not use a standard archive format. You get
both with star and star is even faster than ufsdump.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ragnar Sundblad

On 19 jan 2010, at 20.11, Ian Collins wrote:

> Julian Regel wrote:
>> 
>> Based on what I've seen in other comments, you might be right. 
>> Unfortunately, I don't feel comfortable backing up ZFS filesystems because 
>> the tools aren't there to do it (built into the operating system or using 
>> Zmanda/Amanda).
>> 
> Commercial backup solutions are available for ZFS.
>> I know tape backup isn't sexy, but it's a reality for many of us and it's 
>> not going away anytime soon.
>> 
> True, but I wonder how viable its future is.  One of my clients requires 17 
> LT04 types for a full backup, which cost more and takes up more space than 
> the equivalent in removable hard drives.
> 
> In the past few years growth in hard drive capacities has outstripped tapes 
> to the extent that removable hard drives and ZFS snapshots have become a more 
> cost effective and convenient backup media.

LTO media is still cheaper than equivalent sized disks, maybe a factor 5 or so. 
LTO drivers cost a little, but so do disk shelves. So, now that there is no big 
price issue, there is choice instead. Use it!

Hard drives are good for random access - both restore of individual files and 
partial rewrite. 

Hard drivers aren't faster than tape for data transfer, but they might be 
cheaper to run in parallel and therefore you could potentially gain speed. Hard 
drives have shorter seek time, which may be important.

Hard drives are probably bad for storing for longer times - especially - you 
will never know how long it could be stored before it will fail. A month? 
Probably. A year? Maybe. Five years? Well... Ten years? Probably not. LTO tapes 
are supposed to be able to keep it's data for at least 30 years of stored 
properly. Hard drives are probably best when used online or at least very often.

So - it is wrong to say that one is better or cheaper than the other. They have 
different properties, and could be used to solve different problems.

/ragge s

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-20 Thread Ian Collins

Allen Eastwood wrote:

On Jan 19, 2010, at 22:54 , Ian Collins wrote:

Allen Eastwood wrote:

On Jan 19, 2010, at 18:48 , Richard Elling wrote:

Many people use send/recv or AVS for disaster recovery on the inexpensive
side. Obviously, enterprise backup systems also provide DR capabilities.
Since ZFS has snapshots that actually work, and you can use send/receive
or other backup solutions on snapshots, I assert the "problem" is low priority.

What I have issue with is the idea that no one uses/should use tape any more. There are places for tape and it still has value as a backup device. In many cases in the past, ufsdump, despite it's many issues, was able to restore working OS's, or individual files. Perfect, not by a long shot. But it did get the job done.

As was pointed out earlier, all I needed was a Solaris CD (or network boot) and I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted, same thing…and I can restore just the file(s) I need. And while it's been a few years since I've read the man page on ufsdump, ufsrestore and fssnap, those tools have proven useful when dealing with a downed system.

For a full recovery, you can archive a send stream and receive it back. With
ZFS snapshots, the need for individual file recovery from tape is much reduced.
The backup server I manage for a large client has 60 days of snaps and I can't
remember when they had to go to tape to recover a file.

--
Ian.

Let's see…

For full recovery, I have to zfs send to something, preferably that understands tape (yes, I know I can send to tape directly, but how well does zfs send handle the end of the tape? auto-changers?).
I keep a stream (as a file) of my root pool on a USB stick. It could be
on tape, but root pools are small.

Then for individual file recovery, I have snaphots…which I also have to get on
to tape…if I want to have them available on something other than the boot
devices.

No, just keep the snapshots in place. If a file is lost, just gab it
form the snapshot directory. If the root filesystem is munted, roll
back to the last snapshot.

Now…to recover the entire OS, perhaps not so bad…but that's one tool. And to
recover the one file, say a messed up /etc/system, that's preventing my OS from
booting? Have to get that snapshot where I can use it first…oh and restoring
individual files and not the entire snapshot?

As I said, roll back. Boot form install media, import the root pool,
get the file from a snapshot, or roll back to the last good snapshot,
export and reboot.

At best, it's an unwieldy process. But does it offer the simplicity that
ufsdump/ufsrestore (or dump/restore on how many Unix variants…) did? No way.

It certainly does for file recovery. Do you run incremental dumps every
hour, or every 15 minutes? Periodic snapshots are quick and cheep. As
I said before, careful use of snapshots all be removes the need to
recover files from tape. We have 60 days of 4 hourly and 24 hourly
snapshots in place, so the odds on finding a recent copy of a lost file
are way better than they would be on daily incrementals. I certainly
don't miss the pain of loading a sequence of incrementals to recover
lost data.

So ZFS solves the problems in a different way. For fat-finger recovery,
it's way better than ufsdump/ufsrestore.

A simple, effective dump/restore that deals with all the supported file
systems, can deal with tape or disk, and allows for complete OS restore or
individual file restore, and can be run from a install CD/DVD. As much as I
love ZFS and as many problems as it does solve, leaving this out was a mistake,
IMO.

It possibly was, but it has encouraged us to find better solutions.

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood

On Jan 19, 2010, at 22:54 , Ian Collins wrote:

> Allen Eastwood wrote:
>> On Jan 19, 2010, at 18:48 , Richard Elling wrote:
>> 
>>  
>>> Many people use send/recv or AVS for disaster recovery on the inexpensive
>>> side. Obviously, enterprise backup systems also provide DR capabilities.
>>> Since ZFS has snapshots that actually work, and you can use send/receive
>>> or other backup solutions on snapshots, I assert the "problem" is low 
>>> priority.
>>> 
>>>
>> 
>> 
>> What I have issue with is the idea that no one uses/should use tape any 
>> more.  There are places for tape and it still has value as a backup device.  
>> In many cases in the past, ufsdump, despite it's many issues, was able to 
>> restore working OS's, or individual files.  Perfect, not by a long shot.  
>> But it did get the job done. 
> 
>> As was pointed out earlier, all I needed was a Solaris CD (or network boot) 
>> and I could restore.  Entire OS gone, boot and ufsrestore.  Critical files 
>> deleted, same thing…and I can restore just the file(s) I need.  And while 
>> it's been a few years since I've read the man page on ufsdump, ufsrestore 
>> and fssnap, those tools have proven useful when dealing with a downed system.
>>  
> For a full recovery, you can archive a send stream and receive it back.  With 
> ZFS snapshots, the need for individual file recovery from tape is much 
> reduced.  The backup server I manage for a large client has 60 days of snaps 
> and I can't remember when they had to go to tape to recover a file.
> 
> -- 
> Ian.
> 

Let's see…

For full recovery, I have to zfs send to something, preferably that understands 
tape (yes, I know I can send to tape directly, but how well does zfs send 
handle the end of the tape? auto-changers?).  Then for individual file 
recovery, I have snaphots…which I also have to get on to tape…if I want to have 
them available on something other than the boot devices.

Now…to recover the entire OS, perhaps not so bad…but that's one tool.  And to 
recover the one file, say a messed up /etc/system, that's preventing my OS from 
booting?  Have to get that snapshot where I can use it first…oh and restoring 
individual files and not the entire snapshot?

At best, it's an unwieldy process.  But does it offer the simplicity that 
ufsdump/ufsrestore (or dump/restore on how many Unix variants…) did?  No way.

I suppose it might be fair to say, strictly speaking, that backup/restore 
probably should be dealt with as a "ZFS" issue.  It's kind of ironic that 
Microsoft offers a backup utility and Sun is basically dropping theirs.  It 
might be better to frame the debate somewhere else, but zfs send/receive and 
snapshots are not the same as a built-in OS utility for backing up and 
restoring the OS and OS files.

A simple, effective dump/restore that deals with all the supported file 
systems, can deal with tape or disk, and allows for complete OS restore or 
individual file restore, and can be run from a install CD/DVD.  As much as I 
love ZFS and as many problems as it does solve, leaving this out was a mistake, 
IMO.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Edward Ned Harvey

> Star implements this in a very effective way (by using libfind) that is
> even
> faster that the find(1) implementation from Sun.

Even if I just "find" my filesystem, it will run for 7 hours.  But zfs can
create my whole incremental snapshot in a minute or two.  There is no way
star or any other user-space utility that walks the filesystem can come
remotely close to this performance.  Such performance can only be
implemented at the filesystem level, or lower.


___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Allen Eastwood wrote:

On Jan 19, 2010, at 18:48 , Richard Elling wrote:

As was pointed out earlier, all I needed was a Solaris CD (or network boot) and
I could restore. Entire OS gone, boot and ufsrestore. Critical files deleted,
same thing…and I can restore just the file(s) I need. And while it's been a
few years since I've read the man page on ufsdump, ufsrestore and fssnap, those
tools have proven useful when dealing with a downed system.

For a full recovery, you can archive a send stream and receive it back.
With ZFS snapshots, the need for individual file recovery from tape is
much reduced. The backup server I manage for a large client has 60 days
of snaps and I can't remember when they had to go to tape to recover a file.

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


Joerg Schilling wrote:

Ian Collins  wrote:

  
Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot 
understand them and probably never will be able to understand this format as it

is not well defined for a portable program like star.

  
  
Is that because they are NFSv4 ACLs?  tar uses the formatting routines 
from libacl to archive them.



The correct way to archivbe ACLs would be to put them into extended POSIX tar
attrubutes as star does.

See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.

ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz

The ACL format used by Sun is undocumented..

  

man acltotext

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Mike La Spina

I use zfs send/recv in the enterprise and in smaller environments all time and 
it's is excellent.

Have a look at how awesome the functionally is in this example.

http://blog.laspina.ca/ubiquitous/provisioning_disaster_recovery_with_zfs

Regards,

Mike
-- 
This message posted from opensolaris.org
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood

On Jan 19, 2010, at 18:48 , Richard Elling wrote:

> 
> Many people use send/recv or AVS for disaster recovery on the inexpensive
> side. Obviously, enterprise backup systems also provide DR capabilities.
> Since ZFS has snapshots that actually work, and you can use send/receive
> or other backup solutions on snapshots, I assert the "problem" is low 
> priority.
> 

 What I have issue with is the idea that no one uses/should use tape any more.  
There are places for tape and it still has value as a backup device.  In many 
cases in the past, ufsdump, despite it's many issues, was able to restore 
working OS's, or individual files.  Perfect, not by a long shot.  But it did 
get the job done. 

As was pointed out earlier, all I needed was a Solaris CD (or network boot) and 
I could restore.  Entire OS gone, boot and ufsrestore.  Critical files deleted, 
same thing…and I can restore just the file(s) I need.  And while it's been a 
few years since I've read the man page on ufsdump, ufsrestore and fssnap, those 
tools have proven useful when dealing with a downed system.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Richard Elling

On Jan 19, 2010, at 4:26 PM, Allen Eastwood wrote:

>> Message: 3
>> Date: Tue, 19 Jan 2010 15:48:52 -0500
>> From: Miles Nordin 
>> To: zfs-discuss@opensolaris.org
>> Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability?
>> Message-ID: 
>> Content-Type: text/plain; charset="us-ascii"
>> 
>> I don't think a replacement for the ufsdump/ufsrestore tool is really
>> needed.  From now on, backups just go into Another Zpool.
>> 
>> We need the zfs send stream format commitment (stream format depends
>> only on zfs version, not on zpool version or kernel version), but
>> that's enough.
>> 
>> 
>> If people are really still backing up to tapes or DVD's, just use file
>> vdev's, export the pool, and then copy the unmounted vdev onto the
>> tape or DVD.  The requirement that your backup be staged on a disk
>> isn't an obstacle in the backup direction: Amanda already has this
>> requirement.  Amanda, when I used it, did *not* have this requirement
>> in the restore direction so one would have to keep that in mind: if
>> using tapes, he needs a scratch disk as big as the biggest tape file
>> on the DR site or the development environment or Compliance Extractor
>> Station or wherever the restore is happening.
> 
> 
> While this idea may be fine for a home user…those us of who have customers in 
> the enterprise still have a need for tape backups.
> 
> And some of those enterprises require backup mechanism that can be easily 
> used in a DR situation.
> 
> ufsdump/restore was perfect in that regard.  The lack of equivalent 
> functionality is a big problem for the situations where this functionality is 
> a business requirement.

How quickly we forget ufsdump's limitations :-).  For example, it is not 
supported
for use on an active file system (known data corruption possibility) and 
UFS snapshots are, well, a poor hack and often not usable for backups.
As the ufsdump(1m) manpage says,
 The ufsdump command can only be used on unmounted file  sys-
 tems,  or  those  mounted  read-only.  Attempting  to dump a
 mounted, read-write file system might  result  in  a  system
 disruption  or the inability to restore files from the dump.
 Consider using the fssnap(1M) command to create a file  sys-
 tem  snapshot  if  you  need a point-in-time image of a file
 system that is mounted.

So, if you are taking ufsdumps of an active file system for business 
requirements,
then perhaps you should rethink the solution.

> For example, one customer, local government, requires a backup that can be 
> taken offsite and used in a DR situation.  They have 2 sun servers.  They 
> have very little Solaris knowledge. So whatever I provide them has to be easy 
> and easily documented.  Lack of "zfsdump/restore" means I either have to 
> forgo moving them to ZFS root, or have to add in a third party 
> application…like I'm really going to meet their requirements with Amanda or 
> Backula…

Many people use send/recv or AVS for disaster recovery on the inexpensive
side. Obviously, enterprise backup systems also provide DR capabilities.
Since ZFS has snapshots that actually work, and you can use send/receive
or other backup solutions on snapshots, I assert the "problem" is low priority.

> This lack in Solaris is just plain ludicrous to at least some parts of the 
> business/enterprise world.

Methinks you never read the ufsdump man page? :-)
 -- richard

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Allen Eastwood

> Message: 3
> Date: Tue, 19 Jan 2010 15:48:52 -0500
> From: Miles Nordin 
> To: zfs-discuss@opensolaris.org
> Subject: Re: [zfs-discuss] zfs send/receive as backup - reliability?
> Message-ID: 
> Content-Type: text/plain; charset="us-ascii"
> 
> I don't think a replacement for the ufsdump/ufsrestore tool is really
> needed.  From now on, backups just go into Another Zpool.
> 
> We need the zfs send stream format commitment (stream format depends
> only on zfs version, not on zpool version or kernel version), but
> that's enough.
> 
> 
> If people are really still backing up to tapes or DVD's, just use file
> vdev's, export the pool, and then copy the unmounted vdev onto the
> tape or DVD.  The requirement that your backup be staged on a disk
> isn't an obstacle in the backup direction: Amanda already has this
> requirement.  Amanda, when I used it, did *not* have this requirement
> in the restore direction so one would have to keep that in mind: if
> using tapes, he needs a scratch disk as big as the biggest tape file
> on the DR site or the development environment or Compliance Extractor
> Station or wherever the restore is happening.


While this idea may be fine for a home user…those us of who have customers in 
the enterprise still have a need for tape backups.

And some of those enterprises require backup mechanism that can be easily used 
in a DR situation.

ufsdump/restore was perfect in that regard.  The lack of equivalent 
functionality is a big problem for the situations where this functionality is a 
business requirement.

For example, one customer, local government, requires a backup that can be 
taken offsite and used in a DR situation.  They have 2 sun servers.  They have 
very little Solaris knowledge. So whatever I provide them has to be easy and 
easily documented.  Lack of "zfsdump/restore" means I either have to forgo 
moving them to ZFS root, or have to add in a third party application…like I'm 
really going to meet their requirements with Amanda or Backula…

This lack in Solaris is just plain ludicrous to at least some parts of the 
business/enterprise world.
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Daniel Carosone

There is a tendency to conflate "backup" and "archive", both generally
and in this thread. They have different requirements.  

Backups should enable quick restore of a full operating image with all
the necessary system level attributes. They concerned with SLA and
uptime and outage and data loss window criteria. 

Archives are usually much more concerned with application data, which
may have been specially prepared for the future user of the
archive. Portability is often more important here, both in archive
format but also in data schema within, even at the expense of some
kinds of technical integrity or completeness (e.g. ACLs).

Adding regulatory requirements into the mix further complicates matters.

If one solution can address the union of all requirements, then you're
in luck, but that's not always the case.  Sometimes the best way to
resolve the tension is to save the data differently for different
purposes. 

--
Dan.

pgpfsQPyUEOVE.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Daniel Carosone

On Tue, Jan 19, 2010 at 12:16:01PM +0100, Joerg Schilling wrote:
> Daniel Carosone  wrote:
> 
> > I also don't recommend files >1Gb in size for DVD media, due to
> > iso9660 limitations.  I haven't used UDF enough to say much about any
> > limitations there.
> 
> ISO-9660 supports files up to 8 TB. 

However, some implementations have (or had?) limitations which meant
that they couldn't read such disks.  Some of those platforms even had
problems (or required remembering arcane options and flags for "large
file support") when copying these files back to hard disk.  

Linux was a prime culprit, though not the only one.  I didn't use
linux, but I never knew what might be at hand when needing to read
disks for recovery, so I wanted them to be portable.  I envisaged
using multiple machines to read more disks in parallel, for example.

It may not be as relevant for this use case, nor perhaps for more
modern platforms, but it was a habit I developed earlier for other
kinds of archives. 

I apologise, though, for misattributing this limitation to the
filesystem spec.  I misremembered the reason why I chose the limit.
It was a practical limit, and the practicalities may well have changed
since.

> Do you have a bigger pool?

I certainly don't have a bigger DVD media :-)

(Yes, I know iso9660 also has multivolume support, but I'm similarly
wary of being able to read it back on all platforms, since it's not
encountered often, and there's no need when the source files can be
split to better effect.)

--
Dan.

pgpqa8vKafJ1t.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Miles Nordin

> "ic" == Ian Collins  writes:

 >> I know tape backup isn't sexy, but it's a reality for many of
 >> us and it's not going away anytime soon.

ic> True, but I wonder how viable its future is.  One of my
ic> clients requires 17 LT04 types for a full backup, which cost
ic> more and takes up more space than the

but Ian, aiui SOX requires backups onto a write-once non-eraseable
medium, because of all the discovery shennanigans and
claimed-incompetence around the time of the Worldcomm and Enron
bankruptcy near the start of Bush II.  The law in practice is sort of
like a tiger that chases the herd and eats the sick animal that's
lagging behind, and currently I think the herd is using ``WORM tape''
which is a tape that's made to appear to be append-only by the tape
drive's firmware.  These tapes exist now and must be retained for
seven years, so even if you invented a newfangled gadget meeting the
WORM requirements TODAY so that you can stay in the middle of the herd
without using tape, you still will not get rid of tape for seven
years.

pgp2HKJuL5cLI.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Miles Nordin

> "jk" == Jerry K  writes:

jk> +1 
jk> for zfsdump/zfsrestore

-1

I don't think a replacement for the ufsdump/ufsrestore tool is really
needed.  From now on, backups just go into Another Zpool.

We need the zfs send stream format commitment (stream format depends
only on zfs version, not on zpool version or kernel version), but
that's enough.


If people are really still backing up to tapes or DVD's, just use file
vdev's, export the pool, and then copy the unmounted vdev onto the
tape or DVD.  The requirement that your backup be staged on a disk
isn't an obstacle in the backup direction: Amanda already has this
requirement.  Amanda, when I used it, did *not* have this requirement
in the restore direction so one would have to keep that in mind: if
using tapes, he needs a scratch disk as big as the biggest tape file
on the DR site or the development environment or Compliance Extractor
Station or wherever the restore is happening.

File vdev's have all the wanted characteristics: bitflip resiliency,
checksums, ability to represent all the opaque mysteryfeatures of
inner ZFS's, snapshot/clone efficiency, and importability by future
ZFS kernels ~forever.  It still has the all-or-nothing-restore
downside, but maybe we can live with that.  Those who can't might
stick to spinning-disk backups.

It might be nice to have a ``read-only vdev'' like a mac os compressed
disk image that can occupy the same pool next to a read-write
vdev---this would be neat for livecd's and incrementals---but it's a
neat feature, not something blocking a frequent/unavoidable sysadmin
task.

What's needed IMHO is non-ZFS-specific tools like tar/pax/cpio/rsync
that understand all this new ACL and opaque-fork garbage that
filesystems are growing (and not just Sun's either).  It's needed to
copy between NFSv4 and ZFS, and also to split/combine subdirectories
into separate/common ZFS filesystems without losing any of the
newfangled mysterybaggage.  It has as much to do with shaking corner
cases and awkwardnesses out of the newfangled API's as it does with
actually accomplishing a sysadmin task: the best documentation for the
weird appendages in various modern Unix filesystems would probably be
the tool itself, if we had it.  

Without this kind of tool and the API shakeout that making it would
require, our systems are slowly turning into shitty Windows boxes that
need some black magic proprietary tool like Ghost or PQDI to capture
all the silly out-of-band garbage their installations depend on and/or
accumulate.


pgpnDeqTYeOuV.pgp
Description: PGP signature
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Ian Collins  wrote:

> > Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot 
> > understand them and probably never will be able to understand this format 
> > as it
> > is not well defined for a portable program like star.
> >
> >   
> Is that because they are NFSv4 ACLs?  tar uses the formatting routines 
> from libacl to archive them.

The correct way to archivbe ACLs would be to put them into extended POSIX tar
attrubutes as star does.

See http://cdrecord.berlios.de/private/man/star/star.4.html for a format 
documentation or have a look at ftp://ftp.berlios.de/pub/star/alpha, e.g.
ftp://ftp.berlios.de/pub/star/alpha/acl-test.tar.gz

The ACL format used by Sun is undocumented..

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Joerg Schilling wrote:

Julian Regel wrote:

When I look at the documentation for Zmanda
(http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
it states that the command used to backup a ZFS filesystem depends on whether
backing up ACLs are required (the default is not to(!)). If backing up ACLs are
required - which they are for us - then the native /usr/bin/tar command is
used. Given that /usr/bin/tar doesn't handle sparse files correctly and updates
atime when creating archives, it's not possible to create a complete copy of a
filesystem without making intrusive changes to the structure of the data and/or
metadata.

So it's arguable that ufsdump was a significant improvement over tar, and that
the current archiving solutions for ZFS are actually a step backwards.

From my perspective, Sun really need to create a zfsdump/zfsrestore set of
commands that perform block level backups of a specified filesystem (or
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes
(many of us still use and rely on tape!) etc.

Sun's tar is not able to archive files > 8 GB in a way that can be read on any
other OS unless you use star. This is because Sun's tar does not implement
support for the POSIX.1-2001 extended tar format.

Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot
understand them and probably never will be able to understand this format as it

is not well defined for a portable program like star.

Is that because they are NFSv4 ACLs? tar uses the formatting routines
from libacl to archive them.

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Bob Friesenhahn


On Wed, 20 Jan 2010, Ian Collins wrote:

Commercial backup solutions are available for ZFS.
I know tape backup isn't sexy, but it's a reality for many of us and it's 
not going away anytime soon.


True, but I wonder how viable its future is.  One of my clients requires 17 
LT04 types for a full backup, which cost more and takes up more space than 
the equivalent in removable hard drives.


In the past few years growth in hard drive capacities has outstripped tapes 
to the extent that removable hard drives and ZFS snapshots have become a more 
cost effective and convenient backup media.


In all of these discussions, people seem to forget some very important 
criteria.  That important criteria is the time required for a full 
recovery from backup.


If the company is not able to survive more than a week with total 
disruption of business, but the full restore will take three weeks, 
then the backup has been rendered 100% ineffective.


When planning for recovery from disaster, the time to recover is 
exceedingly important.


When using the backup to filesystem approach, that backup could be 
done to a remote mirrored system (filesystems + hardware) which is 
sufficiently distant to protect against any local disaster.  If there 
is a disaster in the local data center, that system could be 
immediately put on line (assuming adequate connectivity), or that 
system could be loaded on a truck for overnight delivery as a 
replacement to the data center.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


Joerg Schilling wrote:

Ian Collins  wrote:

  

Julian Regel wrote:

Based on what I've seen in other comments, you might be right. 
Unfortunately, I don't feel comfortable backing up ZFS filesystems 
because the tools aren't there to do it (built into the operating 
system or using Zmanda/Amanda).


  

Commercial backup solutions are available for ZFS.



On what archiving programs are these solutions based on?

  
I'm sure they use their own formats.  The only one I have crossed paths 
with is NetVault.


--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Ian Collins  wrote:

> Julian Regel wrote:
> >
> > Based on what I've seen in other comments, you might be right. 
> > Unfortunately, I don't feel comfortable backing up ZFS filesystems 
> > because the tools aren't there to do it (built into the operating 
> > system or using Zmanda/Amanda).
> >
> Commercial backup solutions are available for ZFS.

On what archiving programs are these solutions based on?

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Julian Regel  wrote:

> When I look at the documentation for Zmanda 
> (http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
>  it states that the command used to backup a ZFS filesystem depends on 
> whether backing up ACLs are required (the default is not to(!)). If backing 
> up ACLs are required - which they are for us - then the native /usr/bin/tar 
> command is used. Given that /usr/bin/tar doesn't handle sparse files 
> correctly and updates atime when creating archives, it's not possible to 
> create a complete copy of a filesystem without making intrusive changes to 
> the structure of the data and/or metadata.
>
> So it's arguable that ufsdump was a significant improvement over tar, and 
> that the current archiving solutions for ZFS are actually a step backwards.
>
> > From my perspective, Sun really need to create a zfsdump/zfsrestore set of 
> > commands that perform block level backups of a specified filesystem (or 
> > snapshot) and that preserves ACLs, atime, sparse files, handles multiple 
> > tapes (many of us still use and rely on tape!) etc.

Sun's tar is not able to archive files > 8 GB in a way that can be read on any 
other OS unless you use star. This is because Sun's tar does not implement 
support for the POSIX.1-2001 extended tar format.

Sun's tar also writes ACLs in a way that is 100% non-portable. Star cannot 
understand them and probably never will be able to understand this format as it
is not well defined for a portable program like star.

Star supports to do backups without affecting atime (on Solaris) since 15 years 
already and star supports the withdrawn POSIX draft ACLs in a OS independent 
way. This allows to use star for platform independent ACL support in case you 
are using UFS on Solaris.

Star will in future support NTFS ACLs in a OS independent way. If someone likes 
to contribute, he is of course welcome. As I am currently working on 
cdrtools-3.0-final, star is currently on maintenance only.

What we need for the future is a OS/FS independent program that implements the
needed features.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?


Julian Regel wrote:


Based on what I've seen in other comments, you might be right. 
Unfortunately, I don't feel comfortable backing up ZFS filesystems 
because the tools aren't there to do it (built into the operating 
system or using Zmanda/Amanda).



Commercial backup solutions are available for ZFS.
I know tape backup isn't sexy, but it's a reality for many of us and 
it's not going away anytime soon.


True, but I wonder how viable its future is.  One of my clients requires 
17 LT04 types for a full backup, which cost more and takes up more space 
than the equivalent in removable hard drives.


In the past few years growth in hard drive capacities has outstripped 
tapes to the extent that removable hard drives and ZFS snapshots have 
become a more cost effective and convenient backup media.


What do people with many tens of TB use for backup these days?

--
Ian.

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Julian Regel



>> The beauty of ufsdump/ufsrestore is that because it's bundled with the 
>> operating system, I can perform bare metal recovery using a Solaris DVD and 
>> locally attached tape drive. It's simple and arguably essential for system 
>> administrators.

> Yep. And it was invented because there was no option other than tar
> at the time. Today, there are many very comprehensive backup solutions
> on the market including open source. Some are nicely integrated, such 
> as NexentaStor and Zmanda.

When I look at the documentation for Zmanda 
(http://docs.zmanda.com/Project:Amanda_Enterprise_3.0/ZMC_Users_Manual/Appendix_A:_Backing_Up_and_Restoring_Solaris_ZFS),
 it states that the command used to backup a ZFS filesystem depends on whether 
backing up ACLs are required (the default is not to(!)). If backing up ACLs are 
required - which they are for us - then the native /usr/bin/tar command is 
used. Given that /usr/bin/tar doesn't handle sparse files correctly and updates 
atime when creating archives, it's not possible to create a complete copy of a 
filesystem without making intrusive changes to the structure of the data and/or 
metadata.

So it's arguable that ufsdump was a significant improvement over tar, and that 
the current archiving solutions for ZFS are actually a step backwards.

> From my perspective, Sun really need to create a zfsdump/zfsrestore set of 
> commands that perform block level backups of a specified filesystem (or 
> snapshot) and that preserves ACLs, atime, sparse files, handles multiple 
> tapes (many of us still use and rely on tape!) etc.

> In my crystal ball, I see a divergence away from traditional backup 
> solution approaches. Today you can feel comfortable with backing up
> ZFS file systems because they provide a POSIX file system interface.

Based on what I've seen in other comments, you might be right. Unfortunately, I 
don't feel comfortable backing up ZFS filesystems because the tools aren't 
there to do it (built into the operating system or using Zmanda/Amanda).

I know tape backup isn't sexy, but it's a reality for many of us and it's not 
going away anytime soon.

JR


  ___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Tim Cook

On Tue, Jan 19, 2010 at 11:36 AM, Richard Elling
wrote:

> On Jan 19, 2010, at 1:53 AM, Julian Regel wrote:
> > > When we brought it up last time, I think we found no one knows of a
> > > userland tool similar to 'ufsdump' that's capable of serializing a ZFS
> > > along with holes, large files, ``attribute'' forks, windows ACL's, and
> > > checksums of its own, and then restoring the stream in a
> > > filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'.
> >
> > This has been something I've been working on for the last few days and
> I've not yet found an adequate solution. I've looked at tar, cpio and pax as
> potential tools for backing up a ZFS filesystem to tape, but each has
> limitations. As of today, Sun do not provide the ability to reliably backup
> a Solaris installation without resorting to third party tools. This is
> crazy!
> >
> > The beauty of ufsdump/ufsrestore is that because it's bundled with the
> operating system, I can perform bare metal recovery using a Solaris DVD and
> locally attached tape drive. It's simple and arguably essential for system
> administrators.
>
> Yep. And it was invented because there was no option other than tar
> at the time. Today, there are many very comprehensive backup solutions
> on the market including open source. Some are nicely integrated, such
> as NexentaStor and Zmanda.
>
> > From my perspective, Sun really need to create a zfsdump/zfsrestore set
> of commands that perform block level backups of a specified filesystem (or
> snapshot) and that preserves ACLs, atime, sparse files, handles multiple
> tapes (many of us still use and rely on tape!) etc.
>
> In my crystal ball, I see a divergence away from traditional backup
> solution approaches. Today you can feel comfortable with backing up
> ZFS file systems because they provide a POSIX file system interface.
> Other datasets don't resemble POSIX file systems at all, and I see several
> different types of datasets with the opportunity to become popular with
> the basic ZVol getting a lot of attention lately. The win will go to the
> vendor who can provide the best value for the various datasets in the ZFS
> environment.
>
> > Does anyone know the process to formally request this (we have a support
> contract), or would I be wasting my time?
>
> You can pile onto CR 5004379, want comprehensive backup strategy
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379
>
> I can't answer the latter, but judging by the date of the CR, I wouldn't
> hold my
> breath. Give the fine folks at Zmanda a look.
>
> Also, the ADM project seems to be dead. Unfortunate?
> http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM
>  -- richard
>
>

I'm likely missing something, so please, fill me in.  Aren't they simply
asking for the functionality already provided by NDMP?

-- 
--Tim
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Richard Elling

On Jan 19, 2010, at 1:53 AM, Julian Regel wrote:
> > When we brought it up last time, I think we found no one knows of a
> > userland tool similar to 'ufsdump' that's capable of serializing a ZFS
> > along with holes, large files, ``attribute'' forks, windows ACL's, and
> > checksums of its own, and then restoring the stream in a
> > filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'.
> 
> This has been something I've been working on for the last few days and I've 
> not yet found an adequate solution. I've looked at tar, cpio and pax as 
> potential tools for backing up a ZFS filesystem to tape, but each has 
> limitations. As of today, Sun do not provide the ability to reliably backup a 
> Solaris installation without resorting to third party tools. This is crazy!
> 
> The beauty of ufsdump/ufsrestore is that because it's bundled with the 
> operating system, I can perform bare metal recovery using a Solaris DVD and 
> locally attached tape drive. It's simple and arguably essential for system 
> administrators.

Yep. And it was invented because there was no option other than tar
at the time. Today, there are many very comprehensive backup solutions
on the market including open source. Some are nicely integrated, such 
as NexentaStor and Zmanda.

> From my perspective, Sun really need to create a zfsdump/zfsrestore set of 
> commands that perform block level backups of a specified filesystem (or 
> snapshot) and that preserves ACLs, atime, sparse files, handles multiple 
> tapes (many of us still use and rely on tape!) etc.

In my crystal ball, I see a divergence away from traditional backup 
solution approaches. Today you can feel comfortable with backing up
ZFS file systems because they provide a POSIX file system interface.
Other datasets don't resemble POSIX file systems at all, and I see several
different types of datasets with the opportunity to become popular with
the basic ZVol getting a lot of attention lately. The win will go to the 
vendor who can provide the best value for the various datasets in the ZFS
environment.

> Does anyone know the process to formally request this (we have a support 
> contract), or would I be wasting my time?

You can pile onto CR 5004379, want comprehensive backup strategy
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=5004379

I can't answer the latter, but judging by the date of the CR, I wouldn't hold my
breath. Give the fine folks at Zmanda a look.

Also, the ADM project seems to be dead. Unfortunate?
http://hub.opensolaris.org/bin/view/Project+adm/WhatisADM
 -- richard

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

2010-01-19 Thread Jerry K


+1

for zfsdump/zfsrestore



Julian Regel wrote:



When we brought it up last time, I think we found no one knows of a
userland tool similar to 'ufsdump' that's capable of serializing a ZFS
along with holes, large files, ``attribute'' forks, windows ACL's, and
checksums of its own, and then restoring the stream in a
filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'.


This has been something I've been working on for the last few days and I've not 
yet found an adequate solution. I've looked at tar, cpio and pax as potential 
tools for backing up a ZFS filesystem to tape, but each has limitations. As of 
today, Sun do not provide the ability to reliably backup a Solaris installation 
without resorting to third party tools. This is crazy!

The beauty of ufsdump/ufsrestore is that because it's bundled with the 
operating system, I can perform bare metal recovery using a Solaris DVD and 
locally attached tape drive. It's simple and arguably essential for system 
administrators.

From my perspective, Sun really need to create a zfsdump/zfsrestore set of 
commands that perform block level backups of a specified filesystem (or 
snapshot) and that preserves ACLs, atime, sparse files, handles multiple tapes 
(many of us still use and rely on tape!) etc.

Does anyone know the process to formally request this (we have a support 
contract), or would I be wasting my time?

Thanks

JR

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



  





___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Daniel Carosone  wrote:

> I also don't recommend files >1Gb in size for DVD media, due to
> iso9660 limitations.  I haven't used UDF enough to say much about any
> limitations there.

ISO-9660 supports files up to 8 TB. Do you have a bigger pool?

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Richard Elling  wrote:

> OOB, the default OpenSolaris PATH places /usr/gnu/bin ahead
> of /usr/bin, so gnu tar is "the default."  As of b130 (I'm not running
> an older build currently) the included gnu tar is version 1.22 which
> is the latest as released March 2009 at http://www.gnu.org/software/tar/

This causes Indiana to by default offer tar implementation that creates archives
that are in conflict with the POSIX standard.

Also note that GNU tar is unable to be used as a backup utility:

-   It does neither support ACLs not any other file attributes.

-   It failes even with simples modifications in the original filesystem
and people will become aware of this problem at the time when
they try to restore a backup that will not work in many cases.


> That is my understanding as well.  It seems that the backup vendors
> are moving forward in a more-or-less vendor-specific way. This is
> not necessarily a bad thing, since there are open source solutions.
> But I see the requirements for backups being much more sophisticated
> than ufsdump was 25 years ago.  hmmm... has ufsdump changed over 
> the past 25 years? ;-)

ufsdump did (slightly) change as it now allows you to specify a subdirectory
instead of the whole filesystem.

But what features are you missing?

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Miles Nordin  wrote:

> When we brought it up last time, I think we found no one knows of a
> userland tool similar to 'ufsdump' that's capable of serializing a ZFS
> along with holes, large files, ``attribute'' forks, windows ACL's, and
> checksums of its own, and then restoring the stream in a
> filesystem-agnostic read/write/lseek/... manner like 'ufsrestore'.

Star is such a tool. It privides all the features known from ufsdump
but it is OS and filesystem indepentent. 

Once Sun shows interest in star, there will of course be also support 
for NTFS-ACLs.

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Lassi Tuura  wrote:

> I guess what I am after is, for data which really matters to its owners and 
> which they actually had to recover, did people use tar/pax archives (~ file 
> level standard archive format), dump/restore (~ semi-standard format based on 
> files/inodes) or zfs send/receive (~ file system block level dump), or 
> something else, and why? (Regardless of how these are implemented, hobby 
> scripts or enterprise tools, how they dealt with possible media failure 
> issues, etc.)

star combines the advantages from ufsdump/ufsrestore (true incremental dumps)
with the advantages of a POSIX standard achive format.

Note that star is even at least 30% faster that ufsdump (although ufsdump
reads the raw disk device while star uses the official OS filesystem interface).

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?

Edward Ned Harvey  wrote:

> > I still believe that a set of compressed incremental star archives give
> > you
> > more features.
>
> Big difference there is that in order to create an incremental star archive,
> star has to walk the whole filesystem or folder that's getting backed up,
> and do a "stat" on every file to see which files have changed since the last
> backup run.  If you have a large filesystem, that can take a very long time.

Star implements this in a very effective way (by using libfind) that is even 
faster that the find(1) implementation from Sun.

The big advantage with star is that you get OS and filesystem independent
archives that are compatible with recent POSIX standards and thus can be read 
on any POSIX.1-2001 compliant OS even if you don't have a star binary handy.

> I run incremental zfs send & receive, and it completes typically in a minute
> or two.

Did you make a test with star?

Jörg

-- 
 EMail:jo...@schily.isdn.cs.tu-berlin.de (home) Jörg Schilling D-13353 Berlin
   j...@cs.tu-berlin.de(uni)  
   joerg.schill...@fokus.fraunhofer.de (work) Blog: 
http://schily.blogspot.com/
 URL:  http://cdrecord.berlios.de/private/ ftp://ftp.berlios.de/pub/schily
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] zfs send/receive as backup - reliability?