Re: Multi-machine mirroring choices

2008-07-21 Thread Pete French
 The *big* issue I have right now is dealing with the slave machine going
 down. Once the master no longer has a connection to the ggated devices,
 all processes trying to use the device hang in D status. I have tried
 pkill'ing ggatec to no avail and ggatec destroy returns a message of
 gctl being busy. Trying to ggatec destroy -f panics the machine.

Oddly enough, this was the issue I had with iscsi which made me move
to using ggated instead. On our machines I use '-t 10' as an argument to
ggatec, and this makes it timeout once the connection has been down for
a certain amount of time. I am using gmirror on top, not ZFS, and this
handled the drive vanishing from the mirror quite happily. I haven't
tried it with ZFS, which may not like having the device suddenly dissapear.

-pete.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-21 Thread Sven W



Pete French presumably uttered the following on 07/21/08 07:08:

The *big* issue I have right now is dealing with the slave machine going
down. Once the master no longer has a connection to the ggated devices,
all processes trying to use the device hang in D status. I have tried
pkill'ing ggatec to no avail and ggatec destroy returns a message of
gctl being busy. Trying to ggatec destroy -f panics the machine.


Oddly enough, this was the issue I had with iscsi which made me move
to using ggated instead. On our machines I use '-t 10' as an argument to
ggatec, and this makes it timeout once the connection has been down for
a certain amount of time. I am using gmirror on top, not ZFS, and this
handled the drive vanishing from the mirror quite happily. I haven't
tried it with ZFS, which may not like having the device suddenly dissapear.

-pete.


What I have found is that the master machine will lock up if the slave disappears 
during a large file transfer. I tested this by setting up zpool mirror on the master 
using a ggatec device from the slave. Then I:


pkill'ed ggated on the slave machine.

dd if=/dev/zero of=/data1/testfile2 bs=16k count=8192   [128MB] on the master

The dd command finished and the /var/log/messages showed I/O errors to the slave 
drive as expected. Messages also showed ggatec trying to reconnect every 10 seconds 
(ggatec was started with the -t 10 parameter).


Finally zfs marked the drive unavailable which then allowed me to ggatec destroy -u 
0 without getting the ioctl(/dev/ggctl): Device busy error message. (By the way, 
using ggatec destroy does not kill the ggatec create that created the process to 
begin with, I had to pkill ggatec to get that stop - bug?)


The above behavior would be acceptable for multi-machine mirroring as it would be 
scriptable.


The problem comes with Large writes. I tried to repeat the above with

dd if=/dev/zero of=/data1/testfile2 bs=16k count=32768 [512MB]

which then locks zfs,  and ultimately the system itself. It seems once the write 
size/buffer is full, zfs is unable to fail/unavail the slave drive and the entire 
system becomes unresponsive (cannot even ssh into it).


The bottom line is that without some type of timeout or time to fail (bad I/O to 
fail?) zpool + ggate[cd] seems to be an unworkable solution. This is actually a 
shame as the recover process swapping from master to slave and back again was so 
much cleaner and faster than using gmirror.



Sven
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-16 Thread Aristedes Maniatis


On 15/07/2008, at 3:54 PM, Jeremy Chadwick wrote:

We moved all of our production systems off of using dump/restore  
solely

because of these aspects.  We didn't move to ZFS though; we went with
rsync, which is great, except for the fact that it modifies file  
atimes

(hope you use Maildir and not classic mbox/mail spools...).


We do something similar, except that we use unison rather than rsync.  
This tool is a two way rsync, it deals with collisions and replicating  
files in both directions at once. Very nice. Look for it in the ports  
tree.


This has some advantages for us since we distribute load across  
several machines and have a cluster of machines which all replicate to  
each other. The data is such that collisions are almost never a concern.


Ari Maniatis



--
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001   fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C  5EFA EF6A 7D2E 3E49 102A


___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-16 Thread Andrew Snow


We have deployed an IMAP server running on Cyrus on FreeBSD 6.2, with a 
500GB UFS2 partition mirrored with geom_mirror and geom_gate across a 
dedicated 1gbps link.


It has proven to be very stable and reliable after appropriate tweaking. 
 The uptime of the mirror is usually 1-3 months, sometimes it seems to 
break randomly, possibly because our timeout is too low.  In any case, 
it doesn't take too long to rebuild at about 60mb/s.


(I recently tested the same solution with FreeBSD-7 and found it now 
goes at a full 100mb/s.)


___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Jeremy Chadwick
On Tue, Jul 15, 2008 at 10:07:14AM -0400, Sven Willenberger wrote:
 3) The send/recv feature of zfs was something I had not even considered
 until very recently. My understanding is that this would work by a)
 taking a snapshot of master_data1 b) zfs sending that snapshot to
 slave_data1 c) via ssh on pipe, receiving that snapshot on slave_data1
 and then d) doing incremental snapshots, sending, receiving as in
 (a)(b)(c). How time/cpu intensive is the snapshot generation and just
 how granular could this be done? I would imagine for systems with litle
 traffic/changes this could be practical but what about systems that may
 see a lot of files added, modified, deleted to the filesystem(s)?

I can speak a bit on ZFS snapshots, because I've used them in the past
with good results.

Compared to UFS2 snapshots (e.g. dump -L or mksnap_ffs), ZFS snapshots
are fantastic.  The two main positives for me were:

1) ZFS snapshots take significantly less time to create; I'm talking
seconds or minutes vs. 30-45 minutes.  I also remember receiving mail
from someone (on -hackers?  I can't remember -- let me know and I can
dig through my mail archives for the specific mail/details) stating
something along the lines of over time, yes, UFS2 snapshots take
longer and longer, it's a known design problem.

2) ZFS snapshots, when created, do not cause the system to more or less
deadlock until the snapshot is generated; you can continue to use the
system during the time the snapshot is being generated.  While with
UFS2, dump -L and mksnap_ffs will surely disappoint you.

We moved all of our production systems off of using dump/restore solely
because of these aspects.  We didn't move to ZFS though; we went with
rsync, which is great, except for the fact that it modifies file atimes
(hope you use Maildir and not classic mbox/mail spools...).

ZFS's send/recv capability (over a network) is something I didn't have
time to experiment with, but it looked *very* promising.  The method is
documented in the manpage as Example 12, and is very simple -- as it
should be.  You don't have to use SSH either, by the way[1].

One of the annoyances to ZFS snapshots, however, was that I had to
write my own script to do snapshot rotations (think incremental dump(8)
but using ZFS snapshots).

 I would be interested to hear anyone's experience with any (or all) of
 these methods and caveats of each. I am leaning towards ggate(dc) +
 zpool at the moment assuming that zfs can smartly rebuild the mirror
 after the slave's ggated processes bug out.

I don't have any experience with GEOM gate, so I can't comment on it.
But I would highly recommend you discuss the shortcomings with pjd@,
because he definitely listens.

However, I must ask you this: why are you doing things the way you are?
Why are you using the equivalent of RAID 1 but for entire computers?  Is
there some reason you aren't using a filer (e.g. NetApp) for your data,
thus keeping it centralised?  There has been recent discussion of using
FreeBSD with ZFS as such, over on freebsd-fs.  If you want a link to the
thread, I can point you to it.

I'd like to know why you're doing things the way you are.  By knowing
why, possibly myself or others could recommend solving the problem in a
different way -- one that doesn't involve realtime duplication of
filesystems via network.


[1]: If you're transferring huge sums of data over a secure link (read:
dedicated gigE LAN or a separate VLAN), you'll be disappointed to find
that there is no Cipher=none with stock SSH; the closest you'll get is
blowfish-cbc.  You might be saddened by the fact that the only way
you'll get Cipher=none is via the HPN patches, which means you'll be
forced to install ports/security/openssh-portable.  (I am not a fan of
the overwrite the base system concept; it's a hack, and I'd rather get
rid of the whole base system concept in general -- but that's for
another discussion).  My point is, your overall network I/O will be
limited by SSH, so if you're pushing lots of data across a LAN, consider
something without encryption.

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Kris Kennaway

Jeremy Chadwick wrote:


Compared to UFS2 snapshots (e.g. dump -L or mksnap_ffs), ZFS snapshots
are fantastic.  The two main positives for me were:

1) ZFS snapshots take significantly less time to create; I'm talking
seconds or minutes vs. 30-45 minutes.  I also remember receiving mail
from someone (on -hackers?  I can't remember -- let me know and I can
dig through my mail archives for the specific mail/details) stating
something along the lines of over time, yes, UFS2 snapshots take
longer and longer, it's a known design problem.

2) ZFS snapshots, when created, do not cause the system to more or less
deadlock until the snapshot is generated; you can continue to use the
system during the time the snapshot is being generated.  While with
UFS2, dump -L and mksnap_ffs will surely disappoint you.


a known design problem in the sense of intentional, yes.  They were 
written to support bg fsck, not as a lightweight filesystem feature for 
general use.


Kris
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Pete French
 However, I must ask you this: why are you doing things the way you are?
 Why are you using the equivalent of RAID 1 but for entire computers?  Is
 there some reason you aren't using a filer (e.g. NetApp) for your data,
 thus keeping it centralised?

I am not the roiginal poster, but I am doing something very similar and
can answer that question for you. Some people get paranoid about the
whole single point of failure thing. I originally suggestted that we buy
a filer and have identical servers so if one breaks we connect the other
to the filer, but the response I got was what if the filer breaks?. So
in the end I had to show we have duplicate independent machines, with the
data kept symetrical on them at all times.

It does actually work quite nicely actually - I have an 'active database
machine, and a passive. The opassive is only used if the active fails,
and the drives are run as a gmirror pair with the remote one being mounted
using ggated. It also means I can flip from active to passive when I want
to do an OS upgrade on the active machine. Switching takes a few seconds,
and this is fine for our setup.

So the answer is that the descisiuon was taken out of my hands - but this
is not uncommon, and as a roll-your-own cluster it works very nicely.

-pete.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Oliver Fromme
Sven Willenberger wrote:
  [...]
  1) I have been using ggated/ggatec on a set of 6.2-REL boxes and find
  that ggated tends to fail after some time leaving me rebuilding the
  mirror periodically (and gmirror resilvering takes quite some time). Has
  ggated/ggatec performance and stability improved in 7.0? This
  combination does work, but it is high maintenance and automating it is a
  bit painful (in terms of re-establishing the gmirror and rebuilding and
  making sure the master machine is the one being read from).

First, some problems in ggated/ggatec have been fixed
between 6.2 and 6.3.  Second, you should tune it a little
to improve performance and stability.  The following
reply in an earlier thread is interesting:

http://lists.freebsd.org/pipermail/freebsd-stable/2008-January/039722.html

  2) Noting the issues with ggated/ggatec in (1), would a zpool be better
  at rebuilding the mirror? I understand that it can better determine
  which drive of the mirror is out of sync than can gmirror so a lot of
  the insert rebuild manipulations used with gmirror would not be
  needed here.

I don't think there's much of a difference between gmirror
and a ZFS mirror if used with ggated/ggatec.  Of course,
ZFS has more advantages, like checksumming, snapshots etc.,
but also the disadvantages that it requires considerably
more memory.

Yet another way would be to use DragoFly's Hammer file
system which is part of DragonFly BSD 2.0 which will be
released in a few days.  It supports remote mirroring,
i.e. mirror source and mirror target can run on different
machines.  Of course it is still very new and experimental
(however, ZFS is marked experimental, too), so you probably
don't want to use it on critical production machines.

(YMMV, of course.)

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH  Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

PI:
int f[9814],b,c=9814,g,i;long a=1e4,d,e,h;
main(){for(;b=c,c-=14;i=printf(%04d,e+d/a),e=d%a)
while(g=--b*2)d=h*b+a*(i?f[b]:a/5),h=d/--g,f[b]=d%g;}
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Sven Willenberger
On Tue, 2008-07-15 at 07:54 -0700, Jeremy Chadwick wrote:
 On Tue, Jul 15, 2008 at 10:07:14AM -0400, Sven Willenberger wrote:
  3) The send/recv feature of zfs was something I had not even considered
  until very recently. My understanding is that this would work by a)
  taking a snapshot of master_data1 b) zfs sending that snapshot to
  slave_data1 c) via ssh on pipe, receiving that snapshot on slave_data1
  and then d) doing incremental snapshots, sending, receiving as in
  (a)(b)(c). How time/cpu intensive is the snapshot generation and just
  how granular could this be done? I would imagine for systems with litle
  traffic/changes this could be practical but what about systems that may
  see a lot of files added, modified, deleted to the filesystem(s)?
 
 I can speak a bit on ZFS snapshots, because I've used them in the past
 with good results.
 
 Compared to UFS2 snapshots (e.g. dump -L or mksnap_ffs), ZFS snapshots
 are fantastic.  The two main positives for me were:
 
 1) ZFS snapshots take significantly less time to create; I'm talking
 seconds or minutes vs. 30-45 minutes.  I also remember receiving mail
 from someone (on -hackers?  I can't remember -- let me know and I can
 dig through my mail archives for the specific mail/details) stating
 something along the lines of over time, yes, UFS2 snapshots take
 longer and longer, it's a known design problem.
 
 2) ZFS snapshots, when created, do not cause the system to more or less
 deadlock until the snapshot is generated; you can continue to use the
 system during the time the snapshot is being generated.  While with
 UFS2, dump -L and mksnap_ffs will surely disappoint you.
 
 We moved all of our production systems off of using dump/restore solely
 because of these aspects.  We didn't move to ZFS though; we went with
 rsync, which is great, except for the fact that it modifies file atimes
 (hope you use Maildir and not classic mbox/mail spools...).
 
 ZFS's send/recv capability (over a network) is something I didn't have
 time to experiment with, but it looked *very* promising.  The method is
 documented in the manpage as Example 12, and is very simple -- as it
 should be.  You don't have to use SSH either, by the way[1].

The examples do list ssh as the way of initiating the receiving end; I
am curious as to what the alterative would be (short of installing
openssh-portable and using cipher=no).

 One of the annoyances to ZFS snapshots, however, was that I had to
 write my own script to do snapshot rotations (think incremental dump(8)
 but using ZFS snapshots).

That is what I was afraid of. Using snapshots would seem to involve a
bit of housekeeping. Furthermore, it sounds more suited to a system that
needs periodic rather than constant backing up (syncing).


  I would be interested to hear anyone's experience with any (or all) of
  these methods and caveats of each. I am leaning towards ggate(dc) +
  zpool at the moment assuming that zfs can smartly rebuild the mirror
  after the slave's ggated processes bug out.
 
 I don't have any experience with GEOM gate, so I can't comment on it.
 But I would highly recommend you discuss the shortcomings with pjd@,
 because he definitely listens.
 
 However, I must ask you this: why are you doing things the way you are?
 Why are you using the equivalent of RAID 1 but for entire computers?  Is
 there some reason you aren't using a filer (e.g. NetApp) for your data,
 thus keeping it centralised?  There has been recent discussion of using
 FreeBSD with ZFS as such, over on freebsd-fs.  If you want a link to the
 thread, I can point you to it.

Basically I am trying to eliminate the single point of failure. The
project prior to this had such a failure that even a raid5 setup could
not get out of. It was determined at that point that a single-machine
storage solution would no longer suffice. What I am trying to achieve is
having a slave machine that could take over as the file server in the
event the master machine goes down. This could be something as simple as
the master's network connection going down (CARP to the rescue on the
slave) to a complete failure of the master.

While zfs send/recv sounds like a good option for periodic backups, I
don't think it will fit my purpose. Zpool or gmirror will be a better
fit. I see in posts following my initial post that there is reference to
improvements in ggate[cd] and/or tcp since 6.2 (and I have moved to 7.0
now) so that bodes well. The question then becomes a matter of which
system would be easier to manage in terms of a) the master rebuilding
the mirror in the event the slave goes down or ggate[cd] disconnects and
b) have the slave become the master for serving files and mounting the
drives that were part of the mirror.

Thanks to the other posters, I see others are doing what I am trying to
accomplish and there were some additional tuneables for ggate[cd] that
I had not yet incorporated that were mentioned.

Sven


signature.asc
Description: This is a digitally 

Re: Multi-machine mirroring choices

2008-07-15 Thread Oliver Fromme
Pete French wrote:
  I am not the roiginal poster, but I am doing something very similar and
  can answer that question for you. Some people get paranoid about the
  whole single point of failure thing. I originally suggestted that we buy
  a filer and have identical servers so if one breaks we connect the other
  to the filer, but the response I got was what if the filer breaks?.

You install a filer cluster with two nodes.  Then there is
no single point of failure.

I've done exactly that at customers of my company, i.e.
set up NetApp filer clusters.  Any disk can fail, any shelf
can fail, any filer head can fail.  A complete filer can
fail.  A switch can fail.  The system will keep running
and doing its job.  And yes, we've tested all of that.

Whether filers solve your problems is a different thing.
I just pointed out the answer to the question what if the
filer breaks?.  I'm not a NetApp salesman.  ;-)

Best regards
   Oliver

-- 
Oliver Fromme, secnetix GmbH  Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606,  Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758,  Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart

FreeBSD-Dienstleistungen, -Produkte und mehr:  http://www.secnetix.de/bsd

To this day, many C programmers believe that 'strong typing'
just means pounding extra hard on the keyboard.
-- Peter van der Linden
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Kris Kennaway

Oliver Fromme wrote:


Yet another way would be to use DragoFly's Hammer file
system which is part of DragonFly BSD 2.0 which will be
released in a few days.  It supports remote mirroring,
i.e. mirror source and mirror target can run on different
machines.  Of course it is still very new and experimental
(however, ZFS is marked experimental, too), so you probably
don't want to use it on critical production machines.


Let's not get carried away here :)

Kris
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Pete French
 You install a filer cluster with two nodes.  Then there is
 no single point of failure.

Yes, that would be my choice too. Unfortunately it didn't get
done that way. Mind you, the solution we do have is something
I am actually pretty happy with - it's cheap and does the job.
We never wanted 100% uuptime after all, just something so I
could get stuff back up and running in about 15 minutes with no
loss of data if possible.

Woiuld have been nice to get to play with the NetApp stuff though.

-pete.
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Wesley Shields
On Tue, Jul 15, 2008 at 07:54:26AM -0700, Jeremy Chadwick wrote:
 One of the annoyances to ZFS snapshots, however, was that I had to
 write my own script to do snapshot rotations (think incremental dump(8)
 but using ZFS snapshots).

There is a PR[1] to get something like this in the ports tree.  I have
no idea how good it is but I hope to get it in the tree soon.

http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/125340

-- WXS
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Kris Kennaway

Wesley Shields wrote:

On Tue, Jul 15, 2008 at 07:54:26AM -0700, Jeremy Chadwick wrote:

One of the annoyances to ZFS snapshots, however, was that I had to
write my own script to do snapshot rotations (think incremental dump(8)
but using ZFS snapshots).


There is a PR[1] to get something like this in the ports tree.  I have
no idea how good it is but I hope to get it in the tree soon.

http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/125340


There is also sysutils/freebsd-snapshot (pkg-descr is out of date, it 
supports ZFS too).


I found it more convenient to just write my own tiny script.

Kris
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Matthew Dillon

:Oliver Fromme wrote:
:
: Yet another way would be to use DragoFly's Hammer file
: system which is part of DragonFly BSD 2.0 which will be
: released in a few days.  It supports remote mirroring,
: i.e. mirror source and mirror target can run on different
: machines.  Of course it is still very new and experimental
: (however, ZFS is marked experimental, too), so you probably
: don't want to use it on critical production machines.
:
:Let's not get carried away here :)
:
:Kris

Heh.  I think its safe to say that a *NATIVE* uninterrupted and fully
cache coherent fail-over feature is not something any of us in BSDland
have yet.  It's a damn difficult problem that is frankly best solved
above the filesytem layer, but with filesystem support for bulk mirroring
operations.

HAMMER's native mirroring was the last major feature to go into
it before the upcoming release, so it will definitely be more
experimental then the rest of HAMMER.  This is mainly because it
implements a full blown queue-less incremental snapshot and mirroring
algorithm, single-master-to-multi-slave.  It does it at a very low level,
by optimally scanning HAMMER's B-Tree.  In other words, the kitchen
sink.

The B-Tree propagates the highest transaction id up to the root to
support incremental mirroring and that's the bit that is highly
experimental and not well tested yet.  It's fairly complex because
even destroyed B-Tree records and collapses must propagate a
transaction id up the tree (so the mirroring code knows what it needs
to send to the other end to do comparative deletions on the target).

(transaction ids are bundled together in larger flushes so the actual
B-Tree overhead is minimal).

The rest of HAMMER is shaping up very well for the release.  It's
phenominal when it comes to storing backups.  Post-release I'll be
moving more of our production systems to HAMMER.  The only sticky
issue we have is filesystem-full handling, but it is more a matter
of fine-tuning then anything else.

--

Someone mentioned atime and mtime.  For something like ZFS or HAMMER,
these fields represent a real problem (atime more then mtime).  I'm
kinda interested in knowing, does ZFS do block replacement for
atime updates?

For HAMMER I don't roll new B-Tree records for atime or mtime updates.
I update the fields in-place in the current version of the inode and
all snapshot accesses will lock them (in getattr) to ctime in order to
guarantee a consistent result.  That way (tar | md5) can be used to
validate snapshot integrity.

At the moment, in this first release, the mirroring code does not
propagate atime or mtime.  I plan to do it, though.  Even though
I don't roll new B-Tree records for atime/mtime updates I can still
propagate a new transaction id up the B-Tree to make the changes
visible to the mirroring code.  I'll definitely be doing that for mtime
and will have the option to do it for atime as well.  But atime still
represents a big expense in actual mirroring bandwidth.  If someone
reads a million files on the master then a million inode records (sans
file contents) would end up in the mirroring stream just for the atime
update.  Ick.

-Matt

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Jeremy Chadwick
On Tue, Jul 15, 2008 at 07:10:05PM +0200, Kris Kennaway wrote:
 Wesley Shields wrote:
 On Tue, Jul 15, 2008 at 07:54:26AM -0700, Jeremy Chadwick wrote:
 One of the annoyances to ZFS snapshots, however, was that I had to
 write my own script to do snapshot rotations (think incremental dump(8)
 but using ZFS snapshots).

 There is a PR[1] to get something like this in the ports tree.  I have
 no idea how good it is but I hope to get it in the tree soon.

 http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/125340

 There is also sysutils/freebsd-snapshot (pkg-descr is out of date, it  
 supports ZFS too).

 I found it more convenient to just write my own tiny script.

Thanks for pointing this out -- I had no idea such a port existed!

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Jeremy Chadwick
On Tue, Jul 15, 2008 at 11:47:57AM -0400, Sven Willenberger wrote:
 On Tue, 2008-07-15 at 07:54 -0700, Jeremy Chadwick wrote:
  ZFS's send/recv capability (over a network) is something I didn't have
  time to experiment with, but it looked *very* promising.  The method is
  documented in the manpage as Example 12, and is very simple -- as it
  should be.  You don't have to use SSH either, by the way[1].
 
 The examples do list ssh as the way of initiating the receiving end; I
 am curious as to what the alterative would be (short of installing
 openssh-portable and using cipher=no).

rsh or netcat come to mind.  I haven't tried using either though.

-- 
| Jeremy Chadwickjdc at parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Multi-machine mirroring choices

2008-07-15 Thread Kris Kennaway

Jeremy Chadwick wrote:

On Tue, Jul 15, 2008 at 11:47:57AM -0400, Sven Willenberger wrote:

On Tue, 2008-07-15 at 07:54 -0700, Jeremy Chadwick wrote:

ZFS's send/recv capability (over a network) is something I didn't have
time to experiment with, but it looked *very* promising.  The method is
documented in the manpage as Example 12, and is very simple -- as it
should be.  You don't have to use SSH either, by the way[1].

The examples do list ssh as the way of initiating the receiving end; I
am curious as to what the alterative would be (short of installing
openssh-portable and using cipher=no).


rsh or netcat come to mind.  I haven't tried using either though.



I wouldn't recommend either for the obvious reasons: weak or no 
authentication and integrity protection.  Even if the former is not a 
concern for some reason then the latter should be (your data stream 
could be corrupted in transit and you'd never know until you tried to 
verify or restore the backup).


Kris

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to [EMAIL PROTECTED]