On 09/02/2009, at 1:07 PM, Damien Katz wrote:
On Feb 8, 2009, at 9:24 PM, Chris Anderson wrote:
On Sun, Feb 8, 2009 at 5:54 PM, Damien Katz <[email protected]>
wrote:
It's possible to use MVCC for replication. You'll need to create
special
HTTP command to return you all the documents you are interested in
a single
request, and a special replicator that uses that command and loads
those
documents and writes them to the destination.
This sound a lot like the notification view Damien's been talking
about, where clients can register to be told about database updates
that match particular functions.
The main problem I see with MVCC replication is that if it dies in
the
middle, you might not be able to restart it right where you left off.
That would be a big problem of replicating huge databases.
Everything must come over in one transaction.
You could still do that incrementally e.g. it wouldn't have to load in
a single request. The key is that the replication shows MVCC
boundaries i.e. add a marker in the replication stream to indicate
when you passed an MVCC commit point. The current model would ignore
such markers.- nothing else is required I think. You could even cycle
as long as there were new MVCC states, which would give the same
'includes-updates-as-they-come-in' form of replication, but with
somewhat more consistency. If these restart points were included in
the replication stream, then systems that wanted to allow replication
rollback (see below) could reset the rollback MVCC state when they get
an end-of-MVCC state marker.
If you cared however, and if your application model allowed it, then
you could guarantee consistency by only accepting completed
replications. I imagine you would need to be able to 'undo' an
incomplete replication, which would be a matter of allowing the db to
be rolled back to the MVCC state that was in effect when the
replication started. This would prevent permanent lockup, and I'm sure
you'd want this facility to be enabled/disabled in configuration.
I want to stress that I know this is only useful for a certain class
of use, but I don't think it negatively impacts other uses, so only
those uses that want it would pay for it.
Also, this doesn't resolve the cluster-ACID issue, but I'm confident
there is a solution there that doesn't impact the clustered/non-
exclusive-replication/constant-conflict model.
If you want consistency on the target, you'll have to write lock the
database (not a concept couchdb really has)
I think that's an application-level concept, but I can imagine a patch
that allows it in CouchDB. Still, I'd do that at an application level,
because I'm in user-triggered exclusive replication mode.
until the whole replication completes, or write all the docs in one
big bulk transaction, which won't work for large databases. And
while that transaction is occurring, neither the source database or
target database file itself be compacted (the compaction will take
place, but the old file can't be freed until the full transaction
completes).
Yes, but once again I think there are valid use-cases that allow that
e.g. mine at least.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The trouble with the world is that the stupid are cocksure and the
intelligent are full of doubt.
-- Bertrand Russell
