Re: Multi-DB support within transaction middleware
There's merit in both the responses. Since it's trivial to write your own transaction middleware, I would say the most important thing is to document that the middleware only affects your 'default' database. Then it's up to the developer to decide if they need anything further. On Sun, Nov 13, 2011 at 7:05 PM, Yo-Yo Ma wrote: > Before voting for more magic to the transaction middleware, I'd vote > to remove it altogether. Explicit is surely better than implicit in > this case. The admin already uses commit_on_success, etc. > > On Oct 18, 1:22 pm, David Winterbottom > wrote: > > The current implementation of django.middleware.TransactionMiddleware > does > > not create a transaction for each database, only the default one > > defined by DEFAULT_DB_ALIAS > > -https://code.djangoproject.com/browser/django/trunk/django/db/utils.py. > > Shouldn't the middleware manage a transaction for each database defined > in > > the DATABASES setting? > > > > I discovered this while debugging some caching issues with the > johnnycache > > library, which saves cached querysets when committing. In my case, all > > requests to databases other than default were not being cached. > > > > Happy to supply a patch and tests if this is sensible. > > -- > > *Dr. David Winterbottom* > > Head of Programming > > > > Tangent Labs > > 84-86 Great Portland Street > > London W1W 7NR > > England, UK > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-developers@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > > -- *Dr. David Winterbottom* Head of Programming Tangent Labs 84-86 Great Portland Street London W1W 7NR England, UK -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Multi-DB support within transaction middleware
Before voting for more magic to the transaction middleware, I'd vote to remove it altogether. Explicit is surely better than implicit in this case. The admin already uses commit_on_success, etc. On Oct 18, 1:22 pm, David Winterbottom wrote: > The current implementation of django.middleware.TransactionMiddleware does > not create a transaction for each database, only the default one > defined by DEFAULT_DB_ALIAS > -https://code.djangoproject.com/browser/django/trunk/django/db/utils.py. > Shouldn't the middleware manage a transaction for each database defined in > the DATABASES setting? > > I discovered this while debugging some caching issues with the johnnycache > library, which saves cached querysets when committing. In my case, all > requests to databases other than default were not being cached. > > Happy to supply a patch and tests if this is sensible. > -- > *Dr. David Winterbottom* > Head of Programming > > Tangent Labs > 84-86 Great Portland Street > London W1W 7NR > England, UK -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Multi-DB support within transaction middleware
I'm not sure this is so cut and dry. For two reasons: 1) If a project has a lot of databases configured, created and ending a transaction in all of them for every request/response cycle could get expensive, and following that: 2) This is enough of a change in behavior that it shouldn't just be fixed in the existing middleware without some kind of control over it. Maybe some flag in settings as to which databases the transaction middleware affects, with the default database being the default? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-developers/-/AIhK86iYu1YJ. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: Multi-DB Update
On Wed, Aug 5, 2009 at 2:54 AM, Roman Vorushin wrote:
>
> Alex,
>
>>If anyone has access to an Oracle DB (or any other backend with a
>>custom query object and doesn't mind doing the small bit of porting)
>>and wouldn't mind testing it out in an env with both Oracle and
>>another DB that would be most appreciated
>
> I have access to Oracle database (one of my Django projects uses
> Oracle backend).
>
> Send testing instructions, plz.
>
> Roman Vorushin
> ///
>
> On Aug 4, 10:09 pm, Alex Gaynor wrote:
>> Hey all,
>>
>> This week I implemented the metaclass based solution for backwards
>> compatibility in the Field API that Jacob suggested last week. Then I
>> implemented by first stab at a solution to the Oracle/backend with
>> custom query object problem. However, I don't have access to any of
>> the the DBs I'd need for that, so I'm appealing to django-developers.
>> If anyone has access to an Oracle DB (or any other backend with a
>> custom query object and doesn't mind doing the small bit of porting)
>> and wouldn't mind testing it out in an env with both Oracle and
>> another DB that would be most appreciated. All you'll need to do is
>> have a settings file with something like:
>>
>> DATABASES = {
>> 'default': {
>> 'DATABASE_ENGINE': 'oracle'
>> etc...
>> }
>> 'extra': {
>> 'DATABASE_ENGINE': 'sqlite3',
>> }
>>
>> }
>>
>> And that should be enough.
>>
>> For this week my goal is to get GeoDjango back into working order,
>> since I've broken it rather badly thus far.
>>
>> Thanks,
>> Alex
>>
>> --
>> "I disapprove of what you say, but I will defend to the death your
>> right to say it." -- Voltaire
>> "The people's good is the highest law." -- Cicero
>> "Code can always be simpler than you think, but never as simple as you
>> want" -- Me
>
> >
>
Roman,
Thanks for offering, however Karen Tracey beat you to the punch on
this one :) I've made good progress on ratting out the last of these
bugs.
Alex
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Multi-DB Update
Alex,
>If anyone has access to an Oracle DB (or any other backend with a
>custom query object and doesn't mind doing the small bit of porting)
>and wouldn't mind testing it out in an env with both Oracle and
>another DB that would be most appreciated
I have access to Oracle database (one of my Django projects uses
Oracle backend).
Send testing instructions, plz.
Roman Vorushin
///
On Aug 4, 10:09 pm, Alex Gaynor wrote:
> Hey all,
>
> This week I implemented the metaclass based solution for backwards
> compatibility in the Field API that Jacob suggested last week. Then I
> implemented by first stab at a solution to the Oracle/backend with
> custom query object problem. However, I don't have access to any of
> the the DBs I'd need for that, so I'm appealing to django-developers.
> If anyone has access to an Oracle DB (or any other backend with a
> custom query object and doesn't mind doing the small bit of porting)
> and wouldn't mind testing it out in an env with both Oracle and
> another DB that would be most appreciated. All you'll need to do is
> have a settings file with something like:
>
> DATABASES = {
> 'default': {
> 'DATABASE_ENGINE': 'oracle'
> etc...
> }
> 'extra': {
> 'DATABASE_ENGINE': 'sqlite3',
> }
>
> }
>
> And that should be enough.
>
> For this week my goal is to get GeoDjango back into working order,
> since I've broken it rather badly thus far.
>
> Thanks,
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your
> right to say it." -- Voltaire
> "The people's good is the highest law." -- Cicero
> "Code can always be simpler than you think, but never as simple as you
> want" -- Me
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Multi-DB Status Update
Thanks for you input Justin. I just went on and tried using Django ORM natively on the application it self (to stay under django) and Elixir on all master database interactions. After a little effort, the results were great and I did achieve the behaviour I expected. I'm positively surprised with Elixir, it feels very similar and to be honest, I actually prefer some details of the Model implementation. Anyway I'm implementing the whole structure in a way that allows me to easily change it to Django's own Multi-Database support, when it comes out, so please keep the effort! Ricardo On Jul 29, 2:21 am, Justin Lilly wrote: > On Tue, 28 Jul 2009, Ricardo Santos wrote: > > What is the current status of the multi version support? I mean will > > it be merged with trunk any time soon? > > ... snip ... > > I think this is slated for the 1.2 timecycle. The GSoC istelf ends > sometime in late August, so that's likely the earliest timeframe for > this. That said, it could invariably be later as it may not be in a > usable state. Furthermore, it will likely not be "production ready" > for at least a few weeks after that. > > Hope it helps, > > -justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Status Update
On Tue, 28 Jul 2009, Ricardo Santos wrote: > What is the current status of the multi version support? I mean will > it be merged with trunk any time soon? ... snip ... I think this is slated for the 1.2 timecycle. The GSoC istelf ends sometime in late August, so that's likely the earliest timeframe for this. That said, it could invariably be later as it may not be in a usable state. Furthermore, it will likely not be "production ready" for at least a few weeks after that. Hope it helps, -justin --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Status Update
Hi Alex What is the current status of the multi version support? I mean will it be merged with trunk any time soon? I ask you this because I am at a early stage of designing the architecture of a new project, and this will have to support several clients (separate databases) but connected by a master database (authentication, shared resources, etc). I have achieved this using a process_request middleware that can map to the client, my only problem is changing the database connection from the master database to the client one. Additionally the client will have to replicate some data on the master one, from time to time. I have read a lot of posts on this subject, but I am yet to find a solution that actually works and that I can trust that wont break on a following release, due to using custom Managers with custom connections, feel too much like hacking undocumented features, and therefore totally unsafe. I am considering using SQLalchemy for all the master database interactions, but still use Django ORM for the client ones, as I wish to go away from the Django patterns the least possible and maybe it will make it easier to change it when this multi-database support actually comes out stable. If you could please give me some feedback or suggestion on the path to take, I would gracefully appreciate it. Ricardo --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Week 0 Update
If you recall the conversation between Alex and Simon, it will support this. -justin On May 29, 2009, at 9:29 AM, Marco Louro wrote: > > Hey Alex, > > Not sure if this is the right place, but I guess you're the right > person to ask.. > > Will Multi-BD be flexible enough that it will allow to choose a DB at > runtime, depending on something like the user or the domain and not > tied to models at all? From what I've read that's not possible right > now (don't need to change backends, just database name, password and > server), and it's also not easy. > > > Thanks, > Marco > > > On May 23, 5:05 am, Alex Gaynor wrote: >> Hey all, >> >> GSOC is officially scheduled to start tomorrow (my time), however >> I've take >> the liberty of getting quite a jump on my work (as I described last >> week). >> During this week I got the save using parameter and using queryset >> method, >> these are fairly untested however. The reason for this was after >> updating >> the management commands and testing harness for multiple database >> support >> any fixtures being loaded would be loaded N time (where N is the >> number of >> dbs in your settings.DATABASES setting), so I needed to make sure >> each >> fixture got loaded into each DB, this meant adding the using >> parameter to >> save, which also uses querysets... and it was turtles all the way >> down :) >> >> So while that's basically working, the test suite as a whole is >> still not >> passing (http://paste.pocoo.org/show/118532/is the results for >> me). The >> reason for this (as best I can tell) is that Postgresql runs DDL >> operations >> inside a transaction, and since transaction support isn't actually >> working >> across DB at this point the CREATE TABLE statements never actually >> get >> committed. Which brings us to the problem: updating the >> transaction stuff >> in itself isn't a huge task, and making django use transaction >> stuff where >> necessary isn't tremendously onerous. However, one of >> >> -- >> "I disapprove of what you say, but I will defend to the death your >> right to >> say it." --Voltaire >> "The people's good is the highest law."--Cicero > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Week 0 Update
Hey Alex, Not sure if this is the right place, but I guess you're the right person to ask.. Will Multi-BD be flexible enough that it will allow to choose a DB at runtime, depending on something like the user or the domain and not tied to models at all? From what I've read that's not possible right now (don't need to change backends, just database name, password and server), and it's also not easy. Thanks, Marco On May 23, 5:05 am, Alex Gaynor wrote: > Hey all, > > GSOC is officially scheduled to start tomorrow (my time), however I've take > the liberty of getting quite a jump on my work (as I described last week). > During this week I got the save using parameter and using queryset method, > these are fairly untested however. The reason for this was after updating > the management commands and testing harness for multiple database support > any fixtures being loaded would be loaded N time (where N is the number of > dbs in your settings.DATABASES setting), so I needed to make sure each > fixture got loaded into each DB, this meant adding the using parameter to > save, which also uses querysets... and it was turtles all the way down :) > > So while that's basically working, the test suite as a whole is still not > passing (http://paste.pocoo.org/show/118532/is the results for me). The > reason for this (as best I can tell) is that Postgresql runs DDL operations > inside a transaction, and since transaction support isn't actually working > across DB at this point the CREATE TABLE statements never actually get > committed. Which brings us to the problem: updating the transaction stuff > in itself isn't a huge task, and making django use transaction stuff where > necessary isn't tremendously onerous. However, one of > > -- > "I disapprove of what you say, but I will defend to the death your right to > say it." --Voltaire > "The people's good is the highest law."--Cicero --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Update
On May 2, 12:19 am, Malcolm Tredinnick wrote: > I'd prefer dictionaries to strings, because strings are tough to modify > dynamically -- as has already been demonstrated a few times in Django's > history. That's a pretty strong argument in favour of dictionaries - in which case it might be good to switch the CACHE_BACKEND setting over to using dictionaries instead of a DSN-style string at some point in the future - consistency seems like it would be a good thing here. Cheers, Simon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Update
On Fri, 2009-05-01 at 08:47 -0700, codysacoder wrote: > Why not allow both to work (parse the DSN string if encountered)? That should be an option of last resort. One way to do things, preferably obvious, even if you're not Dutch, and all that jazz. I'd prefer dictionaries to strings, because strings are tough to modify dynamically -- as has already been demonstrated a few times in Django's history. Right now, you have to pass in specific (non-DSN) information to set up a database (in the settings file) and the lower-level API is dictionary based. Introducing DSNs is an option that isn't required. Consistency counts there when trying to choose between two roughly equivalent options. After Simon, Alex, et al have had a discussion at the conference, there'll be a summary and a chance for discussion on this list so we can see if any strong arguments for one option come up there. As you'll see from the thread earlier this week, Simon's reasoning was just that there should be a way to construct things dynamically and he wasn't particularly attached to strings as a concept (so having to arrange for some of the larger conference attendees to help him swim the Vltava at the end of the conference might not be necessary after all). Regards, Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB Update
Why not allow both to work (parse the DSN string if encountered)?
Something like...
DATABASES = {
'default': {'host': 'localhost', 'username': 'user',
'password': 'pass', 'database': 'production'},
'legacy': 'mysql://user:p...@localhost/legacy',
}
On May 1, 2:14 am, Mike Panchenko wrote:
> I would agree with you on the dict issue, given that some of the
> backends have custom settings etc. I also agree that this is not
> really something that matters now - DSNs will need to be parsed into
> dicts anyway, so you might as well implement the dicts first and then
> see what the demand is for DSNs. Supporting both also doesnt seem too
> bad.
>
> Mile.
>
> On Friday, May 1, 2009, Alex Gaynor wrote:
> > Hi all,
>
> > Not a lot of new news or progress. I'll be at EuroDjangoCon all of next
> > week so I may be starting my work there, depending on the status of 1.1.
> > The only new news was Simon and I discussed switching to a DSN string in
> > place of the dict of settings. Personally I'm -1 on that as I find the
> > current setup very readable and doesn't require us to parse strings.
> > However, more importantly, I think it's fairly orthagonal to multidb, and
> > as such should be handled seperately. I think Simon will be at
> > EuroDjangoCon so hopefully I'll have a chance to dicuss it with him there.
>
> > As always, questions, criticisms, curse laden rants, and really good ideas
> > are welcome,
> > Alex
> > --
> > "I disapprove of what you say, but I will defend to the death your right to
> > say it." --Voltaire
> > "The people's good is the highest law."--Cicero
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Multi-DB
On Wed, Apr 29, 2009 at 8:24 PM, Simon Willison wrote: > > On Apr 30, 12:25 am, Alex Gaynor wrote: > > My question would be how is using a DSN and letting those be passed > directly > > to using() any advantage over letting someone pass a dict of those > options > > to using(), or a connection object itself. > > No advantage at all - I'm interested in being able to hand a > dynamically constructed connection to using(), and I don't > particularly mind if it's a DSN, a dictionary or a connection object > I've created already. > > My principle arguments for DSNs are the other two - we already use > them for cache backends, and it's what SQLObject and SQLAlchemy do. > > > Well, my argument for keeping the dictionary maps to, we *already* use it for database settings ;). Also, I personally find DSNs hard to read. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire "The people's good is the highest law."--Cicero --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB
On Apr 30, 12:25 am, Alex Gaynor wrote: > My question would be how is using a DSN and letting those be passed directly > to using() any advantage over letting someone pass a dict of those options > to using(), or a connection object itself. No advantage at all - I'm interested in being able to hand a dynamically constructed connection to using(), and I don't particularly mind if it's a DSN, a dictionary or a connection object I've created already. My principle arguments for DSNs are the other two - we already use them for cache backends, and it's what SQLObject and SQLAlchemy do. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB
On Wed, Apr 29, 2009 at 7:19 PM, Joey Wilhelm wrote: > On Wed, Apr 29, 2009 at 16:03, Simon Willison wrote: > >> 3. (really nutty one this) - a very high scale situation where an >> application is partitioned across hundreds of databases, which each >> one having the same set of tables. This is how WordPress-MU works (as >> used by wordpress.com), with every blog getting its own duplicate set >> of tables - and it's surprisingly effective. >> > > I would have to agree for precisely this reason. I already have a (perhaps > not extremely high scale, but certainly Big Data) project using the > implementation Simon described. Specifically, I have my Django-specific > tables in a PostgreSQL database on one server, and then I have Django > connecting to one of many MS SQL databases on two other machines, dependent > on which client's data is being viewed. > > Obviously this took a good bit of hackery to get it working properly, and > there are still a few issues buried deep within, but I did end up utilizing > DSNs for determining and establishing the appropriate connections. > > This does, however, bring up the point of interacting with multiple > database engines at once, which is not exactly configurable for a DSN. In > general though I give a big +1 for DSNs. > > > > My question would be how is using a DSN and letting those be passed directly to using() any advantage over letting someone pass a dict of those options to using(), or a connection object itself. Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire "The people's good is the highest law."--Cicero --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB
On Wed, Apr 29, 2009 at 16:03, Simon Willison wrote: > 3. (really nutty one this) - a very high scale situation where an > application is partitioned across hundreds of databases, which each > one having the same set of tables. This is how WordPress-MU works (as > used by wordpress.com), with every blog getting its own duplicate set > of tables - and it's surprisingly effective. > I would have to agree for precisely this reason. I already have a (perhaps not extremely high scale, but certainly Big Data) project using the implementation Simon described. Specifically, I have my Django-specific tables in a PostgreSQL database on one server, and then I have Django connecting to one of many MS SQL databases on two other machines, dependent on which client's data is being viewed. Obviously this took a good bit of hackery to get it working properly, and there are still a few issues buried deep within, but I did end up utilizing DSNs for determining and establishing the appropriate connections. This does, however, bring up the point of interacting with multiple database engines at once, which is not exactly configurable for a DSN. In general though I give a big +1 for DSNs. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-DB
On Apr 25, 7:11 am, Alex Gaynor wrote: > Simon uses a new DSN based syntax for specifying DBs, however I find this > pretty akward in the presence of our existing way for defining DB settings. I had a few reasons for suggesting DSNs: * We already use DSN-style syntax for cache backends - it's weird to do it there (for something that isn't actually a database) but not for our database settings. Of course, that inconsistency could be fixed by splitting up the settings for cache backends instead. * SQLObject and SQLAchemy both use DSNs. * If we implement my suggestion that a using() method can take a connection object OR a connection alias string (defined in settings.py) OR a DSN, it's much easier to dynamically construct a database connection at runtime. I'm particularly interested in that last usage scenario. Personally I'm not a big fan of settings.py - over the past four years I've found myself at various times wanting to dynamically alter almost every setting in there - and there are some interesting edge cases where creating a database connection at run time could be useful. Three off the top of my head: 1. A phpMyAdmin style application for administering remote databases, which might allow the user to enter their database details in to an online form 2. A web-based installation mechanism, such as the one used by WordPress 3. (really nutty one this) - a very high scale situation where an application is partitioned across hundreds of databases, which each one having the same set of tables. This is how WordPress-MU works (as used by wordpress.com), with every blog getting its own duplicate set of tables - and it's surprisingly effective. Creating a database connection at runtime doesn't require DSNs of course, provided you can feed your own connection object to any using () clause rather than being tied to using connections that have already been configured in settings.py. > Further a DSN method removes the presence of something like > DATABASE_OPTIONS(unless we start adding query strings onto the end), which > is very useful for things like the postgresql auto commit option, plus I > plan to move the TEST_DATABSE_* settings into DATABASE_OPTIONS, since things > like charset are DB dependent. Again, we use query strings for cache backends so I have no problem with using them for database options. I wouldn't be heartbroken if DSNs weren't used in Django, but the above should at least explain the reasons behind my suggestion. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch
Hi Ed, There are a few people looking at multi-db at the moment. I've had to put my own contributions on hold because I've had rather a hectic personal life lately, but hopefully I'll be able to crack on with it next week or so. There has also been some discussion about rethinking some of the design of multi-db, especially in light of recent changes to the way backends work and forthcoming changes to querying, which is something I'd like to have a look at when I get back into it. Watch this space at the moment it seems! :-) Cheers Ben On 31/10/2007, Ed Summers <[EMAIL PROTECTED]> wrote: > > > On Oct 29, 3:31 pm, "James Bennett" <[EMAIL PROTECTED]> wrote: > > You probably want to start by reading messages here: > > > > http://groups.google.com/group/django-developers/search?q=multi-db&st... > > So it sounds as if the branch is dead, and there are patches being > attached to #4747? I'd be willing to pitch in some time to get the > multi-db code up to snuff, if there is a clear road ahead for what > needs to be done. If there isn't I can live with that too :-) > > //Ed > > > > > -- Regards, Ben Ford [EMAIL PROTECTED] +6281317958862 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch
On Oct 29, 3:31 pm, "James Bennett" <[EMAIL PROTECTED]> wrote: > You probably want to start by reading messages here: > > http://groups.google.com/group/django-developers/search?q=multi-db&st... So it sounds as if the branch is dead, and there are patches being attached to #4747? I'd be willing to pitch in some time to get the multi-db code up to snuff, if there is a clear road ahead for what needs to be done. If there isn't I can live with that too :-) //Ed --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch
On 10/29/07, Ed Summers <[EMAIL PROTECTED]> wrote: > Does anyone know if there are any plans to merge the multi-db branch > [1]. I see in ticket #1142 that things are kind of quiet on the > development side. Did this come up at all during the big spring of > 14-Sep-2007? You probably want to start by reading messages here: http://groups.google.com/group/django-developers/search?q=multi-db&start=0&scoring=d&; -- "Bureaucrat Conrad, you are technically correct -- the best kind of correct." --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: multi-db branch: looking for a new maintainer
Just wanted to put a note on here...that there are people actively using this branch. I've been using it since the beginning of December...and have found it to work fantastically... ( well I did send JP a bug report today ... with a sample project ... when he told me this news ). I do hope in the future to be able to pitch in on this branch and help out ( I did submit a how-to dock in the wiki) ...but am under some serious deadlines right now... so also wouldn't be able to devote the time needed to fully understand everything quickly to take it over... until I meet some deadlines in Feb/March At any rate...I did want to chime in and thank JP for his help when I was getting started...and tell him that this branch of the project is great!! On Jan 24, 3:50 pm, "JP" <[EMAIL PROTECTED]> wrote: > Hi all, > > I'm afraid that I am not going to have time in the future to continue > maintaining and merging up themulti-dbbranch. It's diverged so far > from trunk that any trunk change that touches management.py or models > results in hours of hand-merging and svn detective work... hours that I > just don't have to give right now. > > I'd hate to see the branch die, since I think it will be of use to > people, so I'm hoping that someone else will step forward to take over > ownership, until such a time as the branch is reviewed and merged or > rejected. Of course I'll be more than happy to help the new maintainer > get up to speed on the changes and new features in the branch, and will > always be available to answer questions about it -- as much as time > allows anyway. :) > > Any takers? > > JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Multi-db: Change DatabaseWrappers to no longer be local?
Ivan Sagalaev wrote: > JP wrote: > > Not a dumb question at all. I'll try to explain better by contrasting > > how multi-db handles connections with how connections are kept thread > > local in trunk. > > BTW, I happen to know how it works now since I was among those who was > trying to solve threading issues before Eugene Lazutkin's universal > patch with DatabaseWrapper inherited from local(). Ah! Sorry, I didn't know the history. I'm sort of new in town. :) > I'm asking because I'm worried if something new could break something > working :-) Me too! > This explains it all, thank you! I now actually found this code in > db/backends/__init__.py and from the look of it I agree that inheriting > DatabaseWrapper from local() is no longer needed. Good. So that's one vote for checking in the change to the branch. Anyone else have any thoughts/objections/questions/whatever? Thanks! JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db: Change DatabaseWrappers to no longer be local?
JP wrote: > Not a dumb question at all. I'll try to explain better by contrasting > how multi-db handles connections with how connections are kept thread > local in trunk. BTW, I happen to know how it works now since I was among those who was trying to solve threading issues before Eugene Lazutkin's universal patch with DatabaseWrapper inherited from local(). I'm asking because I'm worried if something new could break something working :-) > In multi-db, all connections (including the default that you can access > as django.db.connection) are managed through a LazyConnectionManager > instance, This explains it all, thank you! I now actually found this code in db/backends/__init__.py and from the look of it I agree that inheriting DatabaseWrapper from local() is no longer needed. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db: Change DatabaseWrappers to no longer be local?
Ivan Sagalaev wrote: > JP wrote: > > Looking back over the multi-db branch today, I realized that it seems > > to be duplicating the thread locality of connections. The connection > > management classes in multi-db django.db manage all connections as > > thread local, and the DatabaseWrapper classes in the backends are all > > subclasses of local. > > > > I may be missing something, but I think the latter locality can be > > safely removed in multi-db. > > I didn't look how exactly multi-db branch manages connection so my > question may be dumb, sorry. If you intend remove the inheritance of > DatabaseWrapper from locals does it mean that when using single DB > DatabaseWrappers will be shared between threads? Not a dumb question at all. I'll try to explain better by contrasting how multi-db handles connections with how connections are kept thread local in trunk. In trunk, django.db.connection is a DatabaseWrapper instance from some backend. DatabaseWrappers all subclass local, so their attributes are thread local, including the self.connection attribute that is the actual db connection. When you try to get a cursor for the first time in a thread, a connection is made based on the current settings and stored in the local attribute. In multi-db, all connections (including the default that you can access as django.db.connection) are managed through a LazyConnectionManager instance, which keeps a thread-local map of connection name to DatabaseWrapper, and is generally accessed through django.db.connections, like: from django.db import connections, _default, connection foo = connections['foo'] connection is connections[_default] # True When a connection is first accessed in a thread, the connection is established based on the appropriate settings (either defaults or a named connection in OTHER_DATABASES) and a DatabaseWrapper is cached in thread local storage. django.db.connection is a lazy proxy that looks up connections[_default] when accessed for the first time in a thread, and caches the result of that call in another local (for efficiency, otherwise we'd be executing a lambda: every time we touched the connection, which would be bad). So each individual connection instance doesn't need to be a local anymore, because it can only be accessed by a lookup into a local, no matter how you get at it. > > DatabaseWrappers don't have to be local > > because each instance can only be accessed in a particular thread > > anyway. > > I don't exactly understand this bit also. Does it mean that each thread > gets its own instance or that threads can access some (shared) instance? Each thread gets its own DatabaseWrapper instance, rather than one DatabaseWrapper (with local attributes) being shared among all threads. Does that make things more clear? JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db: Change DatabaseWrappers to no longer be local?
JP wrote: > Looking back over the multi-db branch today, I realized that it seems > to be duplicating the thread locality of connections. The connection > management classes in multi-db django.db manage all connections as > thread local, and the DatabaseWrapper classes in the backends are all > subclasses of local. > > I may be missing something, but I think the latter locality can be > safely removed in multi-db. I didn't look how exactly multi-db branch manages connection so my question may be dumb, sorry. If you intend remove the inheritance of DatabaseWrapper from locals does it mean that when using single DB DatabaseWrappers will be shared between threads? > DatabaseWrappers don't have to be local > because each instance can only be accessed in a particular thread > anyway. I don't exactly understand this bit also. Does it mean that each thread gets its own instance or that threads can access some (shared) instance? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Replying to myself again... Hacking away at things a bit more, I can see so far only one serious problem with the implementation outlined above. Moving the connection information to model._default_manager.db is going to require changes in django.db.query, specifically in QuerySet._get_sql_clause(), parse_lookup() and lookup_inner(), all of which pass around opts (model._meta) but need access to the model's backend in order to quote names. Is there any objection to changing all of those to pass around the model where needed? Or does anyone have a better idea? I know Adrian is refactoring this module in trunk, so it may be better to put any more multi-db work on the back burner until that's done and I can merge it in. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> OK, thoughts, anyone? When i first heard of multi-db i more thought about support for clustering then moving some apps/models to other databases, which is great, too of course. for example: having one master db and some slave dbs that mirror the master db. requests should go to slaves in some sort of changeable round robin style and changes should go directly to the master db. this scenario popped into my mind the first time i heard of multi-db. is such a scenario supported somehow. i think it will be useful for very large sites and should be kept in mind when the db layer is redesigned. kindly regards daniel --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Replying to myself... here's what I've come up with to explain the problems I see in my current implementation and what I think should be done to fix them. Apologies in advance -- it's quite long. I've implemented a bit of this just to make sure it would work, mainly the basic parts in django.db.__init__.py. Still need to figure out how to come up with some tests for these issues that won't be slow as boiled slugs to run. Anyway, here's the writeup: Django can run as a WSGI app within a container like Paste, where the same app may be run with different settings at the same time in multiple simultaneous threads, and mutiple serial requests within the same thread. Below are listed a few changes that I think are required to make the multi-db branch's settings and connection handling safe for serving in that kind of enviroment. For the purposes of this discussion, assume that the app myproj.myapp is configured within the container with two sets of settings, SA and SB. Imagine that we have a server running threads T1 and T2, and answering requests R1, R2, ... etc. Proposed change: Clear django.db.connections on request finish, make django.db.connections thread-local. Where: django.db.__init__.py Rationale: Settings may change between requests, including what DATABASE_* settings are labeled with a given key in OTHER_DATABASES. django.db.connections is a LazyConnectionManager, so resetting its _connections property between requests will clear named connections for the next request. Of course, R1 may finish on T1 while R2 is running on T2, so we only want to clear the django.db.connections._connections in T1 -- so, django.db.connections._connections must become a thread-local value. Proposed change: Make django.db.connection, .backend, etc proxies Where: django.db.__init__.py Rationale: Apply the same isolation to the default connection and its backend, etc (which are all module-level variables in django.db) as to named connections in django.db.connections. Unfortunately in this case we don't have a convenient dict to clear out at the end of each request. And users may have 'from django.db import connection' anywhere in their code. So django.db.connection, django.db.backend, etc must be thread-local and also cleared at the end of each request, but without rebinding their names. All of these variables are references to attributes in django.db.connection_info, which is a ConnectionInfo instance intialized from the default settings (settings.DATABASE_*). The shortest route to making them request- and thread-safe is to make them instead attributes of a new DefaultConnectionInfoProxy instance, in which each attribute delegates to the same attribute in a ConnectionInfo stored in a thread-local container and reset at the end of each request. Proposed change: Move connection information into Manager.db Where: django.db.models.manager Rationale: Moving the per-model connection information from model._meta to model._default_manager has already been discussed and I think is generally approved -- this is just to flesh out a bit more what that will mean. Since the connection for a model will now be wholly specified by settings, the model's default manager will have to look up the connection to use for the model in settings.OTHER_DATABASES by examining the MODELS entry in each value in that dict for the best match to the model. A manager instance may be shared across threads, so the storage for the current connection must be thread-local. And the same manager may encounter different settings on different requests in the same thread, if (say) R1 in T1 loads SA but R2 also in T1 loads SB, so not only the ConnectionInfo but the whole connection scenario (named or default, name of connection if named, settings to use) must be reset at the end of each request. To do this, I propose making manager.db a descriptor that serves as a thread-local ConnectionInfo proxy. The proxy can be initialized during contribute_to_class and attached only to the instance (not the Manager class). In use, where the current multi-db code has an access pattern like: connection = model._meta.connection The new pattern would be: connection = model._default_manager.db.connection Or in the manager itself: connection = self.db.connection backend = self.db.backend and so on. The model._default_manager.db descriptor will (on first access) initialize and return the correct ConnectionInfo for the current settings state that applies to the manager's model, caching that ConnectionInfo in thread-local storage for the rest of the request and connecting to the request_finished signal to clear the cache. This way, only Managers used during a request would reset at the end of the request. Proposed change: Add a thread-local cache of connections Where: django.db.__init__.py Rationale: Efficiency. Settings may change between requests, but the actual database connection picked out by the tuple (DATABASE_ENGINE
Re: Multi-db branch feedback/questions
[snip most of OTHER_DATABASES proposal] > * ``MODELS`` is a list. Each item in the list may be: > > * ``app_label.module_name`` if a single model should be use this > connection. > * ``app_label`` if the entire app should use this connection > > How does this sound? Good enough for me. Though I weep for all of the tests that are about to break. :) I'm working on a long writeup of the thread/request isolation issues and ideas for how to deal with them that I'll post up later today. I'd like to get some feedback on that stuff before I start hacking away, the changes required look large and tricky. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> What about making Manager callable? It could take a database > identifier, or even an object with connection info, and return a new > Manager instance with the specified database info. This would let you > keep using Model.objects with a default database, or for the case where > multiple databases are not specified. But then you could do something > like this: > > Person.objects(db).filter(...) +1 from me. JP --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 11, 2006, at 8:08 AM, jason pellerin wrote:
> Here's another case that I'd like to see handled gracefully, that I
> think can be handled better with your proposed syntax than mine: one
> app with many models, some of which use a named connection. To me, the
> ideal way to handle that would be to allow entries in APPS to be
> tuples of (app name, [model, model, ...]), like: ('myproject',
> ['ModelA', 'ModelB']) to indicate that the setting applies only to the
> selected models from the app. If apps are keys in DATABASES, you'd
> wind up with keys something like (appname, (model, model, ...)), which
> is pretty grim.
Yeah, that was pretty much my thinking.
After sleeping on it, here's my concrete proposal on how all this
should work. Much of it I think meshes with what's you've already
coded, but there are a few changes:
* All of the current ``DATABASE_*`` remain the same -- they specify
the "main" database. NO change from what you've already god.
* Additionally, the ``DATABASE_*`` settings can be considered
"defaults" for the extra databases (so if you have a bunch of
PostgreSQL databases with the same connection info you only need to
define ``DATABASE_NAME`` for each of the extra DBs. Again, I think
this is the same as what you've already done.
* Other databases are defined in a dict ``OTHER_DATABASES``. I want
to rename this because ``DATABASES`` implies that *all* the databases
go in the dict, instead of just the extra ones. I'm open to other
names, of course, but I want something that makes its secondary
nature pretty obvious.
* The structure of ``OTHER_DATABASES`` is as you've made it, modulo
the name change.
* By default, any app in ``INSTALLED_APPS`` is installed into the
main database. However, each value in ``OTHER_DATABASES`` contains
an ``MODELS`` key which defines which models are installed in that
database.
* ``MODELS`` is a list. Each item in the list may be:
* ``app_label.module_name`` if a single model should be use this
connection.
* ``app_label`` if the entire app should use this connection
How does this sound?
Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
A wrinkle occurred to me last night that I really should have thought
of earlier, and that calls into question my whole connection-handling
setup.
When served as a WSGI app in a container like Paste, settings can be
different on every request. That means, first off, that however we
decide to structure the DATABASES setting, the lookup of app.Model ->
named connection has to be done at request startup. Second, by making
each connection (that is, backend.DatabaseWrapper) take a settings
argument and keep it around, I've broken the current mechanism for
connection request isolation (close the connection at the end of the
request, so it is forced to get a new cursor with possibly new
settings at the start of the next request). Though I'm not sure how
well that method works when app A sees two different DATABASE_ENGINE
settings on two different requests served from the same thread, so
maybe I'm misunderstanding something there.
Anyway, I hope that we can come up with some ideas here for how to
efficiently handle resetting all connection particulars on each
request for both named connections in db.connections, and the default
connection in db.connection (and all of its accessory bits like
backend, etc). I don't have anything concrete to propose right now,
still trying to think it all through. Any ideas, anyone?
> > I'd like to see this done the other way around, which the DATABASES
> > settings defining which *models* are used. I'm picturing something
> > like this::
> >
> > DATABASES = {
> > 'a' : {
> > 'DATABASE_ENGINE' : 'sqlite3',
> > 'DATABASE_NAME': '/whatever/foo.db',
> > 'APPS' : ['myproject', 'yourproject.widget',
> > 'someotherapp']
> > }
> > }
Here's another case that I'd like to see handled gracefully, that I
think can be handled better with your proposed syntax than mine: one
app with many models, some of which use a named connection. To me, the
ideal way to handle that would be to allow entries in APPS to be
tuples of (app name, [model, model, ...]), like: ('myproject',
['ModelA', 'ModelB']) to indicate that the setting applies only to the
selected models from the app. If apps are keys in DATABASES, you'd
wind up with keys something like (appname, (model, model, ...)), which
is pretty grim.
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Trying not to repeat myself too much while arguing for my fast-fading
favorite... :)
> Yeah, you really better, since I don't like it one bit :)
>
> Currently the settings file for lawrence.com has... (/me checks)...
> 48 entries. You mean I've got to duplicate my database settings 48
> times!?
Only if they use 48 different databases. Anything not in DATABASES
would use the default connection. Any database settings that are shared
among > 1 app could be put into a variable ... or the keys in DATABASES
could be tuples.
> I like the idea of keys in ``DATABASES`` being "identifiers" for the
> database connection; this lets something like the following work::
>
> from django import db
> conn = db.get_connection("main")
>
> Which I very much like...
Me too, but I don't know how to reconcile it with configuring which
apps use which settings. I'd be equally happy with:
from django import db
conn = db.get_connection_for('myproj.myapp')
Which is almost as good, with a little extra goodness in not having to
remember anything but the app name to get the connection for that app.
> That makes ``DATABASES`` the canonical location
> for, well, database info, and that's good.
Yes, definitely.
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Another use case to mull around: I'm currently using django for basically everything *but* it's ORM, in large part due to not supporting multiple databases. We have one software package with many nearly-identical databases (for different customers). The models would be the same, but I'd want to decide where to save them at runtime, not as a Meta option. Creating a custom manager for each database would work, but with 10+ databases, it could pollute the namespace, and be tedious to write. > > I posted a couple of example use-cases in response to Malcolm. > > Manually setting ``db_connection`` is ugly but workable; I'll think > > some more about a more elegant way of doing it. What about making Manager callable? It could take a database identifier, or even an object with connection info, and return a new Manager instance with the specified database info. This would let you keep using Model.objects with a default database, or for the case where multiple databases are not specified. But then you could do something like this: Person.objects(db).filter(...) Dan --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
> Under your proposal, I need to write DATABASE_ENGINE and DATABASE_NAME
> once for each application, because there is no way to group the
> information is even one of my applications needs a different database.
> That seems like a lot more configuration (longer config files are harder
> to debug, too).
A few counter-arguments...
If you have five apps that all use the same db, then you don't need
DATABASES at all.
Tuples are hashable:
DATABASES = {
('myproj.fred', 'myproj.ethel', 'myproj.lucy', 'myproj.ricky'): {
'DATABASE_ENGINE': ... }
}
Settings are python files: :)
common_db_settings = { 'DATABASE_ENGINE': ... }
DATABASES = {
'myproj.fred': common_db_settings,
'myproj.ethel': common_db_settings,
'myproj.ricky': common_db_settings,
'myproj.lucy': { 'DATABASE_ENGINE': ... } # she's uncommon
}
Lastly, as I have it now, items in DATABASES inherit from the default
any settings they don't specify themselves, so that should also reduce
config file length when N out of M apps use the same connection.
> I would prefer to get rid of the option of putting it in models
> altogether
You're right, the connection information should belong to the manager.
In my reply to Jacob, I suggested manager.db as the holder for all of
the connection info, which would give you manager.db.connection,
manager.db.backend, etc.
> I am unconvinced. :-)
I have more arguments. Or I will in the morning. :)
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 10, 2006, at 11:02 PM, [EMAIL PROTECTED] wrote:
> That wouldn't matter. Maybe if I take out some ellipses it'll be more
> clear:
>
> DATABASES = {
> 'ellington': { 'DATABASE_NAME': 'ellington',
> 'DATABASE_ENGINE': 'postgresql',
> # DATABASE_USER etc
> },
> 'myproj.myapp': { 'DATABASE_NAME': '/var/db/readonly.db',
> 'DATABSE_ENGINE': 'sqlite3'
> },
> }
>
> The key in DATABASES need not have any relation to the DATABASE_NAME
> setting it references. It seems I'm growing more partial to this as
> the
> simplest solution, so I'd better think of some compelling
> arguments. :)
Yeah, you really better, since I don't like it one bit :)
Currently the settings file for lawrence.com has... (/me checks)...
48 entries. You mean I've got to duplicate my database settings 48
times!?
I like the idea of keys in ``DATABASES`` being "identifiers" for the
database connection; this lets something like the following work::
from django import db
conn = db.get_connection("main")
Which I very much like...
I know you don't like it, but the more I think about it the more I
like the idea of each database stanza providing a list of apps it
should be used for. That makes ``DATABASES`` the canonical location
for, well, database info, and that's good.
Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Jacob Kaplan-Moss wrote:
> On Jul 10, 2006, at 10:09 PM, jason pellerin wrote:
> > Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
> > dropping tables, the branch is fully functional -- please try it out!
>
> I've svn swich'd and I'm fooling with it now...
Cool! I'd suggest starting by reading the tests, othertests/ansi_sql.py
othertests/manager_schema_manipulation.py, and
modeltests/multiple_databases/models.py, if you haven't already.
django/db/backends/ansi/sql.py has the schema manipulation code moved
from django/core/management.py, and there are small changes wherever
db.connection or db.backend was imported, to look those up via the
current model.
> I'd imagine the best way to do it would be to have the ``Manager``
> look up the database connection it's supposed to use (at ``__init__``
> time, probably). That way the lookup would only happen at model
> definition time, not repeatedly for each query. Storing it in
> ``opts`` would be OK, but IMHO a bit ugly since ``opts`` is usually
> reserved for "stuff defined in ``Meta``". Thus you could find out
> which connection a model is using by checking
> ``Model.objects.connection`` (or something other than ``connection``,
> of course).
My underlying opinion is that *all* database-access code should go in
or be referenced through the manager, so I'm all for moving the
connection/connection_info cache there.
If connection/connection_info isn't clear: connection is a
DatabaseWrapper, same as the default django.db.connection.
connection_info is a django.db.ConnectionInfo instance, which
encapsulates all of the metadata about the connection. For instance,
that's where you get at the backend used to create the connection, the
creation module, etc. I'd be quite happy to see the ConnectionInfo at
Model._default_manager.db, which would make the connection
Model._default_manager.db.connection.
> Yeah, I'm not sure I like that... what if I have an app with the same
> name as my database? There's at least one Ellington install I know
> of that has "ellington" in ``INSTALLED_APPS`` and uses a database
> named "ellington"...
That wouldn't matter. Maybe if I take out some ellipses it'll be more
clear:
DATABASES = {
'ellington': { 'DATABASE_NAME': 'ellington',
'DATABASE_ENGINE': 'postgresql',
# DATABASE_USER etc
},
'myproj.myapp': { 'DATABASE_NAME': '/var/db/readonly.db',
'DATABSE_ENGINE': 'sqlite3'
},
}
The key in DATABASES need not have any relation to the DATABASE_NAME
setting it references. It seems I'm growing more partial to this as the
simplest solution, so I'd better think of some compelling arguments. :)
> I'm running a bit dry on other ideas, though...
Hm... an APP_DATABASES setting to map apps to databases? Ugly. Nothing
else is coming to mind, though.
> > Maybe a
> > warning in management.validation if a model references a model that
> > the current settings put until a different connection?
>
> Yeah, that sounds perfect to me.
Excellent. Added to my TODO list on the wiki.
> I posted a couple of example use-cases in response to Malcolm.
> Manually setting ``db_connection`` is ugly but workable; I'll think
> some more about a more elegant way of doing it.
I think Malcolm is right -- if the manager holds the connection, this
problem practically goes away. So long as the manager also implements
save() and delete, that is. Totally made-up example:
class RemoteManager(Manager):
def get_db(self):
return RemoteConnectionInfo()
class Mod:
objects = Manager()
remote = RemoteManager()
# read local, write remote
for m in Mod.objects.all():
Mod.remote.save(m)
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Mon, 2006-07-10 at 22:24 -0500, Jacob Kaplan-Moss wrote: > On Jul 10, 2006, at 10:09 PM, jason pellerin wrote: [...] > >> Finally: will there be syntax to use a specific connection for some > >> operations even if that model's set up to use a different backend? > >> I'd hope so, but I'm not sure I can come up with a good syntax... > > > > Well, you can always set Model._meta.db_connection = 'foo' at any > > time. Or create a manager that pulls the connection to use from > > someplace other than self.model._meta.connection, though that would be > > difficult without some additional abstraction. I'm not quite sure what > > the use of this would be, but that may mean there's a need that I just > > can't see. > > I posted a couple of example use-cases in response to Malcolm. > Manually setting ``db_connection`` is ugly but workable; I'll think > some more about a more elegant way of doing it. One possibility based on what you (Jacob) suggested earlier: if the db connection is a property of the model manager, then it is something you can override by using a different manager. We already have get_query_set() as the "default query set" for a manager. We could have get_db_connection() as the way to get the db connection (or make it a property or an attribute or whatever; the concept is the thing here). Then you can use various managers to get data from different databases. True, you could end having code that looks like this: if preferred_connection = 'first_db': manager = 'first' else: manager = 'second' data = getattr(MyModel, manager).filter(...) which could get old after the first two or three times, although wrapping that in a convenience function (MyModel.get_manager(manager).filter(...)) is not crazy in those cases. In passing, storing the *default* db connection in the Option class doesn't feel too ugly to me: it's a place to store model-specific (as opposed to instance-specific) information. We just happen to populate it mostly from Meta (I'm rewriting history saying it this way, I realise, but does that sound like a reasoning that makes sense). Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 10, 2006, at 10:09 PM, jason pellerin wrote:
> Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
> dropping tables, the branch is fully functional -- please try it out!
I've svn swich'd and I'm fooling with it now...
> This is interesting. How would it work -- look up the APPS in each
> settings entry at model class creation time, and fill in
> opts.db_connection that way? Or do you want to get rid of
> opts.db_connection entirely? I'd prefer to keep it -- for one thing,
> without that attribute around, the work to find the proper connection
> will have to be repeated every time the connection is requested. (If
> it's understood to be more of a connection-name cache rather than the
> prefered point of configuration, that's ok with me.)
Hmm...
I'd imagine the best way to do it would be to have the ``Manager``
look up the database connection it's supposed to use (at ``__init__``
time, probably). That way the lookup would only happen at model
definition time, not repeatedly for each query. Storing it in
``opts`` would be OK, but IMHO a bit ugly since ``opts`` is usually
reserved for "stuff defined in ``Meta``". Thus you could find out
which connection a model is using by checking
``Model.objects.connection`` (or something other than ``connection``,
of course).
> I'm not sure that tucking APPS into the db-definition dict is the
> right thing, though.
Yeah, I'm not really that happy with my syntax; it's just a strawman
to get us thinking about ways to define the connection in the
settings file.
> This may be too "magical", but how about: the
> keys in DATABASES could be app or model names, and in that case,
> matching apps/models will use the connection for that key. Example:
>
> DATABASES = {
># symbolic names are still ok, but you'd have to set them in the
> model
>'a': { ... },
># but specify an app or model and that app or model will use
> this connection
>'myproj.myapp': { ... },
>'thatproj.thatapp.ThatModel': { }
> }
>
> That seems pretty natural and clear to me, at least.
Yeah, I'm not sure I like that... what if I have an app with the same
name as my database? There's at least one Ellington install I know
of that has "ellington" in ``INSTALLED_APPS`` and uses a database
named "ellington"...
I'm running a bit dry on other ideas, though...
> Maybe a
> warning in management.validation if a model references a model that
> the current settings put until a different connection?
Yeah, that sounds perfect to me.
>> Finally: will there be syntax to use a specific connection for some
>> operations even if that model's set up to use a different backend?
>> I'd hope so, but I'm not sure I can come up with a good syntax...
>
> Well, you can always set Model._meta.db_connection = 'foo' at any
> time. Or create a manager that pulls the connection to use from
> someplace other than self.model._meta.connection, though that would be
> difficult without some additional abstraction. I'm not quite sure what
> the use of this would be, but that may mean there's a need that I just
> can't see.
I posted a couple of example use-cases in response to Malcolm.
Manually setting ``db_connection`` is ugly but workable; I'll think
some more about a more elegant way of doing it.
Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Mon, 2006-07-10 at 22:09 -0500, jason pellerin wrote:
> Hi all,
>
> Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
> dropping tables, the branch is fully functional -- please try it out!
>
> > I'd like to see this done the other way around, which the DATABASES
> > settings defining which *models* are used. I'm picturing something
> > like this::
> >
> > DATABASES = {
> > 'a' : {
> > 'DATABASE_ENGINE' : 'sqlite3',
> > 'DATABASE_NAME': '/whatever/foo.db',
> > 'APPS' : ['myproject', 'yourproject.widget',
> > 'someotherapp']
> > }
> > }
> >
> > So each member of APPS is either an app specifier (as you'd pass to
> > models.get_app) or a model specifier (as you'd pass to
> > models.get_model). This way each site can specify different
> > databases for different apps or even individual models.
>
> This is interesting. How would it work -- look up the APPS in each
> settings entry at model class creation time, and fill in
> opts.db_connection that way? Or do you want to get rid of
> opts.db_connection entirely? I'd prefer to keep it -- for one thing,
> without that attribute around, the work to find the proper connection
> will have to be repeated every time the connection is requested. (If
> it's understood to be more of a connection-name cache rather than the
> prefered point of configuration, that's ok with me.)
>
> I'm not sure that tucking APPS into the db-definition dict is the
> right thing, though. This may be too "magical", but how about: the
> keys in DATABASES could be app or model names, and in that case,
> matching apps/models will use the connection for that key. Example:
>
> DATABASES = {
> # symbolic names are still ok, but you'd have to set them in the model
> 'a': { ... },
> # but specify an app or model and that app or model will use this
> connection
> 'myproj.myapp': { ... },
> 'thatproj.thatapp.ThatModel': { }
> }
Under Jacob's proposal, if I have five applications and all the models
in each application use the same connection, I need to write out (at
most), DATABASE_ENGINE and DATABASE_NAME once for each engine/name pair
and I need write the application names once.
Under your proposal, I need to write DATABASE_ENGINE and DATABASE_NAME
once for each application, because there is no way to group the
information is even one of my applications needs a different database.
That seems like a lot more configuration (longer config files are harder
to debug, too).
I would prefer to get rid of the option of putting it in models
altogether, having read Jacob's reasoning: having multiple ways to do
something seems unnecessary here (is it necessary for some reason I
haven't thought of?) and it is encouraging non-portable, difficult to
debug problems when moving applications around. What advantages does it
add that you see?
Jacob's approach still leaves you with a natural key to use when trying
to refer to the connection directly (so keeping DATABASES as a
dictionary, rather than just a list of dictionaries is worthwhile).
>
> That seems pretty natural and clear to me, at least.
I am unconvinced. :-)
Best wishes,
Malcolm
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
On Jul 10, 2006, at 9:55 PM, Malcolm Tredinnick wrote: > I agree with most of your points (the settings one is actually kind of > obvious that it may be backwards now that you mention it). > Inter-database relations would be nice to have at some point, but it > could be a post-first-release kind of feature. I agree it would be cool, but it would be (possibly horrifically) slow, and (for me at least) fringes on YAGNI. > On Mon, 2006-07-10 at 21:38 -0500, Jacob Kaplan-Moss wrote: > [...] >> Finally: will there be syntax to use a specific connection for some >> operations even if that model's set up to use a different backend? >> I'd hope so, but I'm not sure I can come up with a good syntax... > > What sort of use cases do you have in mind for this? Constructing > inter-db relations by hand? Not that in particular; I was thinking of things like: - importing data from another database - "feeding" data between two Django installations - saving data from a feed to multiple databases in one script (All of these are real-world problems I've found a way around in the past). > Thinking of some of his earlier mails, I think Jason already had in > mind > some way of getting the specific connections via db.backend, since we > would need that for manually constructing SQL in any case. But the way > you phrased the above query makes me think you have some trickier > use in > mind. I'm not sure it's really that complicated, frankly. Just being able to use the ORM against to database simultaneously would be a huge win for me. Jacob --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
Hi all,
Thanks, Jacob. And yes please, more eyes are badly needed. Aside from
dropping tables, the branch is fully functional -- please try it out!
> I'd like to see this done the other way around, which the DATABASES
> settings defining which *models* are used. I'm picturing something
> like this::
>
> DATABASES = {
> 'a' : {
> 'DATABASE_ENGINE' : 'sqlite3',
> 'DATABASE_NAME': '/whatever/foo.db',
> 'APPS' : ['myproject', 'yourproject.widget',
> 'someotherapp']
> }
> }
>
> So each member of APPS is either an app specifier (as you'd pass to
> models.get_app) or a model specifier (as you'd pass to
> models.get_model). This way each site can specify different
> databases for different apps or even individual models.
This is interesting. How would it work -- look up the APPS in each
settings entry at model class creation time, and fill in
opts.db_connection that way? Or do you want to get rid of
opts.db_connection entirely? I'd prefer to keep it -- for one thing,
without that attribute around, the work to find the proper connection
will have to be repeated every time the connection is requested. (If
it's understood to be more of a connection-name cache rather than the
prefered point of configuration, that's ok with me.)
I'm not sure that tucking APPS into the db-definition dict is the
right thing, though. This may be too "magical", but how about: the
keys in DATABASES could be app or model names, and in that case,
matching apps/models will use the connection for that key. Example:
DATABASES = {
# symbolic names are still ok, but you'd have to set them in the model
'a': { ... },
# but specify an app or model and that app or model will use this connection
'myproj.myapp': { ... },
'thatproj.thatapp.ThatModel': { }
}
That seems pretty natural and clear to me, at least.
> Second, there's no mention about how relations will work across
> databases. I'm sure we *could* emulate them, but I'd suggest
> throwing a nice error if you try to reference a model that's in
> another database.
There's a *little* mention in the tests. :) My preference is not to
enforce this sort of thing, but let the database do it when needed.
Mainly because one of the big use cases for me is the ability to use a
local sqlite cache on a spread of webservers. The caches are published
from an admin db, but only the appropriate public tables are sent; so
there may be models and relations that appear to cross connections on
the public side but share a connection on the admin side. Maybe a
warning in management.validation if a model references a model that
the current settings put until a different connection?
> Finally: will there be syntax to use a specific connection for some
> operations even if that model's set up to use a different backend?
> I'd hope so, but I'm not sure I can come up with a good syntax...
Well, you can always set Model._meta.db_connection = 'foo' at any
time. Or create a manager that pulls the connection to use from
someplace other than self.model._meta.connection, though that would be
difficult without some additional abstraction. I'm not quite sure what
the use of this would be, but that may mean there's a need that I just
can't see.
> OK, thoughts, anyone?
What he said. :)
JP
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Multi-db branch feedback/questions
I agree with most of your points (the settings one is actually kind of obvious that it may be backwards now that you mention it). Inter-database relations would be nice to have at some point, but it could be a post-first-release kind of feature. On Mon, 2006-07-10 at 21:38 -0500, Jacob Kaplan-Moss wrote: [...] > Finally: will there be syntax to use a specific connection for some > operations even if that model's set up to use a different backend? > I'd hope so, but I'm not sure I can come up with a good syntax... What sort of use cases do you have in mind for this? Constructing inter-db relations by hand? Thinking of some of his earlier mails, I think Jason already had in mind some way of getting the specific connections via db.backend, since we would need that for manually constructing SQL in any case. But the way you phrased the above query makes me think you have some trickier use in mind. Malcolm --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
