Sage Weil wrote:
On Fri, 31 Jul 2009, Jeff Garzik wrote:
Sage Weil wrote:
Hi Jeff,
Do you still plan to replace bdb (and it's replication) with a something
based on paxos? I'm considering replacing the Ceph monitors (which
currently implement paxos, but in a very ceph-specific way) with cld if it
can meet the basic requirements.
What I'd kind of like to see is a clean implementation of a paxos
library--one that leaves out message transport and storage--to build a
replicated write-ahead log. And then a separate library for handling the
database/namespace served up by cld (be it regular files, bdb, whatever)
that leaves replication up to paxos. It looks like Google ended up doing
something similar with Chubby (see
http://labs.google.com/papers/paxos_made_live.html).
Does this sound like the direction you guys are heading in?
You mean something like http://linux.yyz.us/misc/paxreg.c ? :)
Yeah, for starters. I'm thinking of the larger problem of integrating of
master elections, add leasing and timeouts (to avoid querying peers for
reads), and so forth to make core paxos usable in a practical environment.
And the glue to bind it to the database (snapshotting and log trimming,
catch-up, etc.).
You have outlined my ideas for CLD version 2.0, essentially: create a
libpaxos and libpaxos_db, and use those in CLD.
So, 100% agreed...
Unfortunately that is a lower priority for me than hammering out a rock
solid CLD <-> cldc network protocol, and getting out a version 1.0 of
the CLD service with _some_ form of solid, working replication and
master fail-over.
I figured, for CLD version 1.0, db4 already went through the pain of
debugging a replicated database. Avoiding that myself would help get
CLD up and running much more rapidly.
I would be very happy to take PAXOS database patches from others, though ;-)
Most importantly, from the view of a CLD client (libcldc user), CLD will
provide the necessary guarantees today. When CLD switches to native PAXOS,
the CLD client API will not change at all. So, the switchover should be
transparent from the client's point of view.
Of course. :)
I'm going to look a bit more closely at what it'll take to moving ceph to
cld, then. Among other things, it'll mean part of cldc in the kernel, but
should be a net architectural improvement.
Cool! A couple kernel-related comments:
* some operations involved in master-discovery and master-failover, most
notably DNS SRV lookups, you probably want to do in userspace
* libcldc is intentionally written such that you should be able to use
lib/cldc.c in embedded applications (such as the kernel), and
successfully ignore related modules cldc-udp.c and cldc-dns.c.
* as such, I am happy to take patches that gets lib/cldc.c as close as
possible to your kernel version of cldc core.
* Not strictly kernel-related, but I do need to adjust the license of
CLD (and chunkd and tabled) libraries to be more friendly to linking
with other applications. Presumably LGPL...
Jeff
--
To unsubscribe from this list: send the line "unsubscribe hail-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
