On Mon, 4 Apr 2011 15:17:22 +0530, Anurag Priyam <[email protected]> wrote: > Mapping Queries to Shards > ------------------------- > > Use a vBucket[1][2] style partitioning scheme coupled with libhashkit[3] for a > pluggable hashing solution. > > vBucket offers greater flexibility over ketama in partitioning data. It > follows > a two stage mapping: > a. a hashing fucntion maps data to virtual buckets > b. virtual buckets statically map to nodes
vBuckets seem to be a name used elsewhere for the current method inside NDB as well. NDB chose 240 as the number to map to as it's quite factorable. e.g. if you had 2 machines, 120 partitions each. This makes going up to 240 machines rather easy, you just relocate a partition. The big benefit is the reduction of round-trips. considering that machines running clients connect regularly, caching the sharding information on them is certainly not out of the question (it wouldn't be large). The mapping can also change, and could quite easily be implemented for moving a shard to a new machine. We'd just need a way in the server to return that a database is read only (attempt r/w op on a shard, get back "currently read only") while doing the migration. After migration, ideally we could use an error saying "shard has relocated" at which point the client could update its mapping and connect to the correct server. I say "database" but in future this could be CATALOG (and this would enforce the no-cross-shard queries rule). This kind of scheme has the advantage of being completely ignorant of schema. In fact, the sharding keys could be anything. I'm not convinced that having to be able to map each row in each table to a particular shard key is a worthwhile approach. The only real way to do this would be foreign keys, and there's no guarantee that a) schema design will include proper foreign keys b) foreign keys are wanted to be used (they do have a perf impact). I also don't like a mechanism that would require another round trip to find out which server to connect to in order to run the actual query (it also pretty much just moves your scaling and availability problem around rather than solving it at all). I also think that you shouldn't try to solve every problem in the scope of this project (e.g. migrating shards/vbuckets, dealing with r/o replicas). Getting the first steps solid and efficient can be enough work. Setting good steps along the way is useful not only for setting goals, but that as each step is completed (including the first), something useful enters the tree. hope this helps, -- Stewart Smith _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
