I've been doing a little reading on multi-db code an wiki. You've basically been tackling problem #3 (different data types and engines) - which I didn't care about at all. That's good, I guess.
The way I handle database connections is just by having a connection pool of different connection objects alive at the same time and create new cursors on the connections I need. Since I have only implemented the most simple SELECT FROM WHERE for one and many rows I haven't worried too much about commit and rollback and stuff like that. So I don't get all of your code and why you need to use threadlocals and stuff like that. The basic thing I do when a Django Model is sharded to different server is this: 1. I write a get_shards classfunction for every model that does some logic and returns a list of one or more shard objects that have a link to this shard's connection. 2. Ask the model class on which shards it is, this returns a list of shard objects. 3. For each shard object I get a new database cursor from the connection which lives in a seperate connection object for every shard (I'm not perfectly sure if this is thread safe) 4. For each of those cursors I repeat the query Django wanted to run. Then I try to stich the responses together the best I can. You guys obviously know your code better than me. Should I start re- writing my code (necessary after queryset-refactoring) based on your patch? Jan On Jul 8, 4:04 pm, "Ben Ford" <[EMAIL PROTECTED]> wrote: > Hi Jan, > > It sounds like you've made great progress. We have an informal trac and hg > repo set up at trac and hg dot woe-beti.de respectively. you're more than > welcome to add your documentation there! Let me know if you want an hg repo > tp play with too and I'll sort it out for you. > > Cheers, > Ben > > 2008/7/8 Jan Oberst <[EMAIL PROTECTED]>: > > > > > > > Hi guys, > > > I've been heavily swamped with work for college, so I missed this > > thead and the few others on multiple databases. Sorry. > > > I have implemented a proof-of-concept database scaling solution for > > Django. It tackles all kind of scaling issues I have seen in Django. > > It's purpose is mainly to find out if we could scale up Django at all. > > I didn't worry too much about syntax and the way it's supposed to > > integrate into Django - I just hacked away in Django code to make it > > work the fastest possible way I could think of. > > > The solution covers the largest part of Simon's #2 problem. I added a > > few attributes and config parameters to the ORM so you can decide > > which models are hosted on which server. One model can be hosted on 20 > > servers with the actual location depending on a foreign key value. > > > We're using it to store data for different groups on different servers > > for a more horizontal scaling. For example if a photo got a ForeignKey > > to group A it will be routed to server 15 because of some logic. > > > You can also route objects 1-1000 to server 1 and 1001-2000 to server > > 2. > > > I have also added database denormalization, caching foreign key > > querysets to the DB, bulk prefetching, in-model privacy checks and a > > few other things. > > > A large percentage of the stuff probably isn't suitable for Django- > > trunk. Most of it tackles quite specific and hard scaling issues, but > > I guess there's a way to build it more modular and make it work for > > more people. After all I'm new to Django-developers and also to > > opening up my work. > > > If some of you are interested in the code and would benefit from it I > > would be more than happy to share. > > > Just posting a big pile of code probably won't help you too much, so I > > thought I'd write a few lines documentation about each part and post > > them here. Does that sound reasonable? > > > Jan --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
