Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sun, Dec 6, 2009 at 12:35 AM, Simon Willison  wrote:
> On Dec 5, 4:20 pm, Russell Keith-Magee  wrote:
>> Trust me - I don't want to do mindless busy work. However, we need to
>> have some sort of answer for the admin interface - Django's admin is a
>> big selling point for Django for some people, so we can't really
>> introduce a huge new feature, and then say "but you can't use it in
>> admin". I'm interested in making the least intrusive change that is
>> possible without hamstringing future multi-db interfaces.
>
> It strikes me that the admin issue should be solvable entirely in
> django.contrib.admin without any changes to the multidb code itself.
> Right now you can get most of the job done using a ModelAdmin
> subclass:
>
> from django.contrib import admin
> from blog.models import Entry
>
> class EntryAdmin(admin.ModelAdmin):
>    def save_model(self, request, obj, form, change):
>        obj.save(using='otherdb')
>
>    def queryset(self, request):
>        return Entry.objects.using('otherdb').all()
>
> admin.site.register(Entry, EntryAdmin)
>
> I haven't tested the above so it's probably missing a few cases
> (save_fieldsets for example perhaps) but if we document it I think
> it's a good enough solution for the moment. Even if we need to
> refactor ModelAdmin a bit to ensure the right hooks are available it
> still shouldn't be a massive change.

I want to poke around this code over the next couple of days. I
suspect that you're probably right - the set of changes won't be too
significant. However, if it is possible to provide a simple
configuration option so that end-users don't have to write boilerplate
save_model() and queryset() definitions, I think that's worth looking
into.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sun, Dec 6, 2009 at 1:09 AM, Tobias McNulty  wrote:
> On Sat, Dec 5, 2009 at 10:51 AM, Russell Keith-Magee
>  wrote:
>>
>> I don't grant that proposition at all. The admin interface serves as a
>> working example demonstrating that you don't need to use settings to
>> define the way models are used.
>
> Okay.  Do you grant the proposition that "we will (not necessarily in Django
> 1.2) need a project-level way to specify the default database(s) to use for
> queries to a given model," whether it is settings-based or not?

Possibly. :-)

>> At this point, I'm prognosticating because I haven't actually written
>> any code for this - but I don't think the using argument to Site()
>> would necessarily have to be deprecated. In a single-database admin,
>> the using argument tells you exactly which database is to be used; if
>> we update admin to have a fully multi-db interface in the future, the
>> value of the using argument could easily be interpreted as the
>> "default" database that is displayed.
>
> Maybe I'm just being obtuse, but there are a couple issues with the admin
> approach to which I can't see a resolution yet:
> * Assuming the above proposition is true, the behavior of the API when the
> global and admin-level configurations conflict is not at all intuitive

I'm not sure how you can make that assertion given that neither global
nor admin-level configurations for multi-db exist yet.

Might I humbly suggest that we stop speculating about the possible
limitations of a theoretical admin implementation until such time as
an implementation actually exists?

> Simon's idea seems like a reasonable (and already supported?) workaround for
> those who need to modify the admin to use a different database in specific
> cases.

Agreed. All I'm talking about is making it easier to do what Simon has
suggested, but without the need for boilerplate ModelAdmin
definitions. I'm hoping to spend the next couple of days fiddling with
this problem. Once we've got something concrete to argue about, I'll
report back.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sun, Dec 6, 2009 at 4:16 AM, subs...@gmail.com  wrote:
> Oh, I see from a later message by Alex that Meta.using was removed.
>
> -1!

There's a very good reason why this was removed. It isn't a model
level property. Consider - what if contrib.auth.User had a Meta
using='foo' property? If this were the case, you wouldn't be able to
reuse the contrib.auth.User model without defining a 'foo' database.
This would singlehandedly hobble the ability reuse Django
applications.

On top of that, the capability provided by Meta.using can easily be
reproduced with a custom manager. I've given an example of how this
would work in this thread.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[subs...@gmail.com]

Oh, I see from a later message by Alex that Meta.using was removed.

-1!

-S

On Dec 5, 3:12 pm, "subs...@gmail.com"  wrote:
> Isn't 'database' going to be an option in a model's Meta? In this
> situation, is admin going to attempt to do something different?
>
> -S
>
> On Dec 4, 9:18 am, Nan  wrote:
>
> > > 1) Ignore the problem. Admin works on the default database, but
> > > nowhere else. This is certainly less than ideal, but it would be
> > > sufficient for master/slave setups.
>
> > > 2) Use a separate admin deployment for each database. We add a 'using'
> > > argument to admin.Site(), and append .using() the queries that the
> > > admin site issues.
>
> > > 3) Modify the admin list views so that they take a ?using=db GET
> > > argument; also add a pulldown to make it easy to switch to a different
> > > database.
>
> > > (2) should be a fairly minor change; (3) would be a little more
> > > effort, but shouldn't be impossible. There is also the need for some
> > > mechanics to check whether a table is actually present on a database -
> > > just because auth is in INSTALLED_APPS doesn't mean it's been
> > > synchronized to a particular database.
>
> > > I'm open to any other suggestions - and for any offers to help out :-)
>
> > Just thinking of one more option for this -- what about specifying the
> > DB on the ModelAdmin level rather than the admin.Site level?

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[subs...@gmail.com]

Isn't 'database' going to be an option in a model's Meta? In this
situation, is admin going to attempt to do something different?

-S

On Dec 4, 9:18 am, Nan  wrote:
> > 1) Ignore the problem. Admin works on the default database, but
> > nowhere else. This is certainly less than ideal, but it would be
> > sufficient for master/slave setups.
>
> > 2) Use a separate admin deployment for each database. We add a 'using'
> > argument to admin.Site(), and append .using() the queries that the
> > admin site issues.
>
> > 3) Modify the admin list views so that they take a ?using=db GET
> > argument; also add a pulldown to make it easy to switch to a different
> > database.
>
> > (2) should be a fairly minor change; (3) would be a little more
> > effort, but shouldn't be impossible. There is also the need for some
> > mechanics to check whether a table is actually present on a database -
> > just because auth is in INSTALLED_APPS doesn't mean it's been
> > synchronized to a particular database.
>
> > I'm open to any other suggestions - and for any offers to help out :-)
>
> Just thinking of one more option for this -- what about specifying the
> DB on the ModelAdmin level rather than the admin.Site level?

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

On Sat, Dec 5, 2009 at 10:51 AM, Russell Keith-Magee  wrote:
>
> I don't grant that proposition at all. The admin interface serves as a
> working example demonstrating that you don't need to use settings to
> define the way models are used.
>

Okay.  Do you grant the proposition that "we will (not necessarily in Django
1.2) need a project-level way to specify the default database(s) to use for
queries to a given model," whether it is settings-based or not?

At this point, I'm prognosticating because I haven't actually written
> any code for this - but I don't think the using argument to Site()
> would necessarily have to be deprecated. In a single-database admin,
> the using argument tells you exactly which database is to be used; if
> we update admin to have a fully multi-db interface in the future, the
> value of the using argument could easily be interpreted as the
> "default" database that is displayed.


Maybe I'm just being obtuse, but there are a couple issues with the admin
approach to which I can't see a resolution yet:

* Assuming the above proposition is true, the behavior of the API when the
global and admin-level configurations conflict is not at all intuitive

* It seems like confining every admin site to a single database would lead
to needless partitioning of the admin interface for what might otherwise be
a single, cohesive site.

Simon's idea seems like a reasonable (and already supported?) workaround for
those who need to modify the admin to use a different database in specific
cases.

Cheers,
Tobias
-- 
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
(919) 951-0052
http://www.caktusgroup.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 5, 4:20 pm, Russell Keith-Magee  wrote:
> Trust me - I don't want to do mindless busy work. However, we need to
> have some sort of answer for the admin interface - Django's admin is a
> big selling point for Django for some people, so we can't really
> introduce a huge new feature, and then say "but you can't use it in
> admin". I'm interested in making the least intrusive change that is
> possible without hamstringing future multi-db interfaces.

It strikes me that the admin issue should be solvable entirely in
django.contrib.admin without any changes to the multidb code itself.
Right now you can get most of the job done using a ModelAdmin
subclass:

from django.contrib import admin
from blog.models import Entry

class EntryAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
obj.save(using='otherdb')

def queryset(self, request):
return Entry.objects.using('otherdb').all()

admin.site.register(Entry, EntryAdmin)

I haven't tested the above so it's probably missing a few cases
(save_fieldsets for example perhaps) but if we document it I think
it's a good enough solution for the moment. Even if we need to
refactor ModelAdmin a bit to ensure the right hooks are available it
still shouldn't be a massive change.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 11:33 PM, Waldemar Kornewald
 wrote:
> Hi Russell,
>
> On Sat, Dec 5, 2009 at 2:58 PM, Russell Keith-Magee
>  wrote:
>> On Sat, Dec 5, 2009 at 9:05 PM, Waldemar Kornewald  
>> wrote:
>>> On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
>>>  wrote:
 The idea of using a function that returns a single string but does
 other processing is a novel approach, and one that I hadn't
 considered. However, I'm not sure I'm especially fond of the idea of
 requiring imports in a settings file, and the syntax you propose is
 quite verbose. I'll have to think about this a bit. Thanks for the
 suggestion, though.
>>>
>>> Shouldn't all this be more abstract? For our non-relational branch we
>>> thought about something more flexible like a list of query proxies
>>> that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
>>> and that can intercept query execution and instead run their own query
>>> code.
>>
>> I'm afraid I don't see the connection between what you're describing,
>> and the problem we actually have.
>>
>> The problem is determining whether model X is available on database Y.
>> This is only required for metaprogramming purposes - so, for example,
>> syncdb knows which models to synchronize onto a particular database,
>> or an admin interface knows which models can be shown to the end user.
>>
>> As best as I can make out, you're addressing the problem that I've
>> said we aren't addressing - that of presenting a useful end-user API
>> for tasks like master/slave. If I'm mistaken, feel free to correct me
>> - preferably with some sample code to demonstrate what you're talking
>> about.
>
> Yes and no. Aren't both tasks (model location and
> sharding&master/slave API) highly connected?

Well, yes, they are connected - there is a one way dependency. We can
have a solution for model locations that doesn't require a solution
for public API, but not the other way around.

> We don't know what the API for sharding, master/slave, etc. will look
> like, but I have the impression that we're already defining its
> settings format and planning changes that won't be necessary with
> Django 1.3, anymore.

Others in this thread might be convinced of the need for a setting,
but I really *don't* want to introduce a setting at this point in
proceedings.

I asked for feedback, and concern about the absence of a setting
providing database-model assignments is useful feedback. I've
explained why there isn't such a setting in the code at the moment. If
someone is able to come up with an elegant solution to the problem,
I'm interested in hearing it. If I'm discussing settings formats, it's
only in the context of exploring options that we may not have
previously considered.

As I've mentioned a few times now, I'm acutely aware of the fact that
this patch doesn't address public API issues for common use cases, and
I don't want to introduce features in this release that we need to
deprecate in later releases.

> For example, as my suggestion shows, the admin
> interface does not have to (and must not!) know about the multi-db
> setup because that can be fully abstracted behind an sql.Query proxy
> API (actually, it must be abstracted that way because there can be
> lots of different ways to do sharding and you don't want high-level
> code to deal with those details).

Trust me - I don't want to do mindless busy work. However, we need to
have some sort of answer for the admin interface - Django's admin is a
big selling point for Django for some people, so we can't really
introduce a huge new feature, and then say "but you can't use it in
admin". I'm interested in making the least intrusive change that is
possible without hamstringing future multi-db interfaces.

> I am just concerned that you'll waste time on implementing something
> that will get removed in 1.3 shortly thereafter just because multi-db
> isn't ready for real-world use-cases, yet. I'd rather have a temporary
> very simple hack that binds a model to a single DB and that doesn't
> require any changes to the admin interface itself and code that uses
> the admin UI. This could indeed be specified in the settings as was
> already suggested. But it should be clear that this is just a
> temporary hack and 1.3 will provide a clean solution.

In a previous message in this thread, I described a temporary hack to
bind a model to a specific database - override get_query_set() on the
model manager. It's not a perfect solution for all situations, but it
is a workable hack, and it doesn't require any settings, temporary or
otherwise.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 11:15 PM, Tobias McNulty  wrote:
> On Sat, Dec 5, 2009 at 8:58 AM, Russell Keith-Magee
>  wrote:
>> As best as I can make out, you're addressing the problem that I've
>> said we aren't addressing - that of presenting a useful end-user API
>> for tasks like master/slave. If I'm mistaken, feel free to correct me
>> - preferably with some sample code to demonstrate what you're talking
>> about.
>
> Yes - it sounds to me like Waldemar might be talking about the
> "registration/callback API to allow you control database assignment on
> a per-query basis" that you mention above.  I hadn't thought about
> this before, but I like the idea and I think it'll be crucial in
> implementing some of the more complex partitioning cases.
>
> It seems like it'll be necessary in addition to some sort of
> settings-based map of what apps/models go in what databases.

I don't grant that proposition at all. The admin interface serves as a
working example demonstrating that you don't need to use settings to
define the way models are used.

I'm not saying that this is necessarily the best way to solve the
problem - just that a setting-based map isn't an absolute given. There
are many viable alternatives, all of which we can explore when we
actually have working multi-db plumbing in the field.

> Quick question: will the admin.Site method be useful in addition to
> whatever more global method we come up with for designating what
> tables go in what databases, or will it be obsolete (and potentially
> deprecated) at that point?  I can't see a reason that one would want
> to modify the database(s) the admin uses irrespective of the global
> settings.

At this point, I'm prognosticating because I haven't actually written
any code for this - but I don't think the using argument to Site()
would necessarily have to be deprecated. In a single-database admin,
the using argument tells you exactly which database is to be used; if
we update admin to have a fully multi-db interface in the future, the
value of the using argument could easily be interpreted as the
"default" database that is displayed.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Waldemar Kornewald]

Hi Russell,

On Sat, Dec 5, 2009 at 2:58 PM, Russell Keith-Magee
 wrote:
> On Sat, Dec 5, 2009 at 9:05 PM, Waldemar Kornewald  
> wrote:
>> On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
>>  wrote:
>>> The idea of using a function that returns a single string but does
>>> other processing is a novel approach, and one that I hadn't
>>> considered. However, I'm not sure I'm especially fond of the idea of
>>> requiring imports in a settings file, and the syntax you propose is
>>> quite verbose. I'll have to think about this a bit. Thanks for the
>>> suggestion, though.
>>
>> Shouldn't all this be more abstract? For our non-relational branch we
>> thought about something more flexible like a list of query proxies
>> that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
>> and that can intercept query execution and instead run their own query
>> code.
>
> I'm afraid I don't see the connection between what you're describing,
> and the problem we actually have.
>
> The problem is determining whether model X is available on database Y.
> This is only required for metaprogramming purposes - so, for example,
> syncdb knows which models to synchronize onto a particular database,
> or an admin interface knows which models can be shown to the end user.
>
> As best as I can make out, you're addressing the problem that I've
> said we aren't addressing - that of presenting a useful end-user API
> for tasks like master/slave. If I'm mistaken, feel free to correct me
> - preferably with some sample code to demonstrate what you're talking
> about.

Yes and no. Aren't both tasks (model location and
sharding&master/slave API) highly connected?

We don't know what the API for sharding, master/slave, etc. will look
like, but I have the impression that we're already defining its
settings format and planning changes that won't be necessary with
Django 1.3, anymore. For example, as my suggestion shows, the admin
interface does not have to (and must not!) know about the multi-db
setup because that can be fully abstracted behind an sql.Query proxy
API (actually, it must be abstracted that way because there can be
lots of different ways to do sharding and you don't want high-level
code to deal with those details).

I am just concerned that you'll waste time on implementing something
that will get removed in 1.3 shortly thereafter just because multi-db
isn't ready for real-world use-cases, yet. I'd rather have a temporary
very simple hack that binds a model to a single DB and that doesn't
require any changes to the admin interface itself and code that uses
the admin UI. This could indeed be specified in the settings as was
already suggested. But it should be clear that this is just a
temporary hack and 1.3 will provide a clean solution.

Bye,
Waldemar Kornewald

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

On Sat, Dec 5, 2009 at 8:58 AM, Russell Keith-Magee
 wrote:
> As best as I can make out, you're addressing the problem that I've
> said we aren't addressing - that of presenting a useful end-user API
> for tasks like master/slave. If I'm mistaken, feel free to correct me
> - preferably with some sample code to demonstrate what you're talking
> about.

Yes - it sounds to me like Waldemar might be talking about the
"registration/callback API to allow you control database assignment on
a per-query basis" that you mention above.  I hadn't thought about
this before, but I like the idea and I think it'll be crucial in
implementing some of the more complex partitioning cases.

It seems like it'll be necessary in addition to some sort of
settings-based map of what apps/models go in what databases.

Quick question: will the admin.Site method be useful in addition to
whatever more global method we come up with for designating what
tables go in what databases, or will it be obsolete (and potentially
deprecated) at that point?  I can't see a reason that one would want
to modify the database(s) the admin uses irrespective of the global
settings.

Cheers,
Tobias
-- 
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
(919) 951-0052
http://www.caktusgroup.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 9:05 PM, Waldemar Kornewald  wrote:
> On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
>  wrote:
>> The idea of using a function that returns a single string but does
>> other processing is a novel approach, and one that I hadn't
>> considered. However, I'm not sure I'm especially fond of the idea of
>> requiring imports in a settings file, and the syntax you propose is
>> quite verbose. I'll have to think about this a bit. Thanks for the
>> suggestion, though.
>
> Shouldn't all this be more abstract? For our non-relational branch we
> thought about something more flexible like a list of query proxies
> that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
> and that can intercept query execution and instead run their own query
> code.

I'm afraid I don't see the connection between what you're describing,
and the problem we actually have.

The problem is determining whether model X is available on database Y.
This is only required for metaprogramming purposes - so, for example,
syncdb knows which models to synchronize onto a particular database,
or an admin interface knows which models can be shown to the end user.

As best as I can make out, you're addressing the problem that I've
said we aren't addressing - that of presenting a useful end-user API
for tasks like master/slave. If I'm mistaken, feel free to correct me
- preferably with some sample code to demonstrate what you're talking
about.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Waldemar Kornewald]

On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
 wrote:
> The idea of using a function that returns a single string but does
> other processing is a novel approach, and one that I hadn't
> considered. However, I'm not sure I'm especially fond of the idea of
> requiring imports in a settings file, and the syntax you propose is
> quite verbose. I'll have to think about this a bit. Thanks for the
> suggestion, though.

Shouldn't all this be more abstract? For our non-relational branch we
thought about something more flexible like a list of query proxies
that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
and that can intercept query execution and instead run their own query
code.

For example, a proxy could emulate unsupported DB features like
select_related for non-relational DBs. You could also use this for
selecting the connection for each query (e.g., sharding by pk) and for
spreading a single query across multiple DBs at the same time (e.g.,
if you want to find all users born in 1980 you'll have to run this
query on all DBs of a sharded users table).

The point is, QuerySet and Model shouldn't care about those details.
They should only provide a high-level abstraction to the DB that is as
expressive and simple as possible. The details can be implemented via
add-ons, so everyone can map the DB abstraction to his real DB setup.

Bye,
Waldemar Kornewald

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 4:07 PM, Johannes Dollinger
 wrote:
>
> Am 05.12.2009 um 06:36 schrieb Russell Keith-Magee:
>> [...]
>>> What if the admin was instead fixed
>>> by providing facilities for the more general case outlined above?
>>>
>>> What would this look like?  I'm picturing another setting (bleh) that
>>> maps apps and/or models to specific databases.  Name choices aside,
>>> this might look something like:
>>>
>>> DATABASE_USING = {
>>>     'django.contrib.auth': 'default',
>>>     'myapp': 'database1',
>>>     'myapp2': {
>>>         'Model1': 'database2',
>>>         'Model2': 'database3',
>>>     }
>>> }
>>>
>>> The admin could automatically take advantage of this setting to
>>> determine what database to use for a given model.
>>
>> Alex, Joseph Kocherhans and I discussed this exact idea at Djangocon.
>> You are correct that it has a lot of potential uses - not only the
>> admin, but also loaddata/dumpdata, syncdb, and anywhere else that an
>> iteration over models is required.
>>
>> However, it's a little bit more complicated than you make out.
>>
>> This sort of setting is very easy to give as a 10 line example, but in
>> practice, this isn't what you will require - you will effectively need
>> to duplicate the contents of INSTALLED_APPS. I have projects in the
>> wild with dozens of entries in INSTALLED_APS - all running on a single
>> database. Writing a DATABASE_USING setting like the one you describe
>> would be extremely cumbersome and annoying.
>>
>> So, we could hijack INSTALLED_APPS to represent the idea of database
>> deployment, using tuples/dictionaries rather than strings to define
>> how apps are deployed. However, this comes at the cost of backwards
>> compatibility for anyone iterating over INSTALLED_APPS.
>> [...]
>
> Let me propose a different colored pattern. It's backwards compatible,
> dry, feels like the urlconf approach and may evolve into a fix for
> #3591:
>
> INSTALLED_APPS = (
>        app('django.contrib.auth', using=...),
>        app('myapp'),
>        app('myapp2', models=(
>                model('Model1', using=...),
>                model('Model2', using=...),
>        ),
> )
>
> Where `app()` would look like this:
>
> APP_USING = {}
>
> def app(path, **kwargs):
>        if 'using' in kwargs:
>                APP_USING[path] = kwargs['using']
>        ...
>        return path
>
> The downside: It would require an import in settings.py. And the
> global dict is not exactly beautiful - but the price for bc.

The idea of using a function that returns a single string but does
other processing is a novel approach, and one that I hadn't
considered. However, I'm not sure I'm especially fond of the idea of
requiring imports in a settings file, and the syntax you propose is
quite verbose. I'll have to think about this a bit. Thanks for the
suggestion, though.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Johannes Dollinger]

Am 05.12.2009 um 06:36 schrieb Russell Keith-Magee:
> [...]
>> What if the admin was instead fixed
>> by providing facilities for the more general case outlined above?
>>
>> What would this look like?  I'm picturing another setting (bleh) that
>> maps apps and/or models to specific databases.  Name choices aside,
>> this might look something like:
>>
>> DATABASE_USING = {
>> 'django.contrib.auth': 'default',
>> 'myapp': 'database1',
>> 'myapp2': {
>> 'Model1': 'database2',
>> 'Model2': 'database3',
>> }
>> }
>>
>> The admin could automatically take advantage of this setting to
>> determine what database to use for a given model.
>
> Alex, Joseph Kocherhans and I discussed this exact idea at Djangocon.
> You are correct that it has a lot of potential uses - not only the
> admin, but also loaddata/dumpdata, syncdb, and anywhere else that an
> iteration over models is required.
>
> However, it's a little bit more complicated than you make out.
>
> This sort of setting is very easy to give as a 10 line example, but in
> practice, this isn't what you will require - you will effectively need
> to duplicate the contents of INSTALLED_APPS. I have projects in the
> wild with dozens of entries in INSTALLED_APS - all running on a single
> database. Writing a DATABASE_USING setting like the one you describe
> would be extremely cumbersome and annoying.
>
> So, we could hijack INSTALLED_APPS to represent the idea of database
> deployment, using tuples/dictionaries rather than strings to define
> how apps are deployed. However, this comes at the cost of backwards
> compatibility for anyone iterating over INSTALLED_APPS.
> [...]

Let me propose a different colored pattern. It's backwards compatible,  
dry, feels like the urlconf approach and may evolve into a fix for  
#3591:

INSTALLED_APPS = (
app('django.contrib.auth', using=...),
app('myapp'),
app('myapp2', models=(
model('Model1', using=...),
model('Model2', using=...),
),
)

Where `app()` would look like this:

APP_USING = {}

def app(path, **kwargs):
if 'using' in kwargs:
APP_USING[path] = kwargs['using']
...
return path

The downside: It would require an import in settings.py. And the  
global dict is not exactly beautiful - but the price for bc.

This is not really an argument for a settings based solution (+0). But  
if there will be one, don't make it nested tuples/dicts - the  
existance of tuple literals in python does not make s-expressions a  
natural representation of configuration.
__
Johannes

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 4:42 AM, Yuri Baburov  wrote:
> Hi all,
>
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
>
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.

As I noted in my explanatory notes for the call for feedback, this
iteration of multi-db is the 'porcelain but not the polished seat'
version of multi-db. You can direct a contrib.auth query to any
database you want with this iteration. This is a necessary
prerequisite for any more sophisticated API for multi-db.

I fully expect that the 1.3 timeframe will see many proposals for how
to present a more useful API to end users of multidb. Controlling and
configuring Managers will be a big part of that API.

Alex's position is all about avoiding a rush into the second phase. We
all want a pretty multi-db interface. We just don't need to rush the
development of that interface. However, we do need to establish that
the plumbing work that the pretty interface will use is solid and
compatible with existing code.

> Alex, you just don't get it seriously yet.

Yuri - for your benefit: this particular turn of phrase and tone is
quite condescending, and could easily be interpreted as offensive. I
certainly hope that this is just a combination of crossed wires,
enthusiasm, and a language barrier.

Yours
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Sat, Dec 5, 2009 at 3:34 AM, Tobias McNulty  wrote:
> The very first "Requirement"/ideal feature on the wiki page reads:
>
> "Different models/applications living on different databases, for
> example a 'blog' application on db1 and a forum application on db2.
> This should include the ability to assign a different database to an
> existing application without modifying it, e.g. telling Django where
> to keep the django.contrib.auth.User table. "
>
> That sounds perfect - having a way to modify in what database a table
> exists on a per-model or per-app basis, without making any changes to
> the app itself (so I can choose to put reusable app A in database X
> and reusable app B in database Y).
>
> I don't see anything in the docs about if and how this type of
> partitioning is supported.

The underlying idea of multidb is that you can direct any query to any
database you choose. If you want to force a particular model to a
particular database by default, then you install a default manager
that directs queries for that model onto that database of choice:

class MyManager(models.Manager):
def get_query_set(self):
return super(MyManager, self).get_query_set().using('other')

You then use syncdb to synchronize applications onto the databases
where you want them.

Now - I fully acknowledge that this is problematic for models like
contrib.auth, as you don't have the ability to install a default
manager. As I noted in my explanatory notes for the call for feedback,
this iteration is the 'porcelain but not the polished seat' version of
the code. You can direct a contrib.auth query to any database you want
with this iteration; in 1.3, I expect that Django will gain some form
of registration/callback API to allow you control database assignment
on a per-query basis.

> From what I understand, it has been
> proposed that this be implemented for the admin specifically through
> ModelAdmin and/or at the admin.Site level, because in the current
> iteration of the code there is no way to use the admin for anything
> other than the default database.

The idea about having Site bound to a database is consistent with the
view of contrib.admin as an Adminstration interface for your database.
If you have two databases, having two adminstration interfaces isn't a
huge logical jump.

Of course, it would be nice to be able to switch between databases
inside a single admin interface, and I don't see any reason that this
couldn't be done - it will just take more effort to implement.

> To me, specifying the database in the ModelAdmin or admin.Site seems
> arbitrary and potentially limiting: For any reusable app on the
> market, depending on the value of a particular setting in your urls.py
> or admin.py file is a Bad Idea.

I don't follow that logic. Admin sites are deployed into urls.py by
the end user - who knows exactly what databases are available, and
what models are available on those databases.

The only time this would be a problem for Site is if a reusable app
included an admin deployment as part of it's urls.py - but I can't
think of any reusable app that does this.

It is arguably true of ModelAdmin. However, remember if you don't like
the ModelAdmin for contrib.auth, you can unregister that ModelAdmin
and install your own.

> What if the admin was instead fixed
> by providing facilities for the more general case outlined above?
>
> What would this look like?  I'm picturing another setting (bleh) that
> maps apps and/or models to specific databases.  Name choices aside,
> this might look something like:
>
> DATABASE_USING = {
>     'django.contrib.auth': 'default',
>     'myapp': 'database1',
>     'myapp2': {
>     'Model1': 'database2',
>     'Model2': 'database3',
>     }
> }
>
> The admin could automatically take advantage of this setting to
> determine what database to use for a given model.

Alex, Joseph Kocherhans and I discussed this exact idea at Djangocon.
You are correct that it has a lot of potential uses - not only the
admin, but also loaddata/dumpdata, syncdb, and anywhere else that an
iteration over models is required.

However, it's a little bit more complicated than you make out.

This sort of setting is very easy to give as a 10 line example, but in
practice, this isn't what you will require - you will effectively need
to duplicate the contents of INSTALLED_APPS. I have projects in the
wild with dozens of entries in INSTALLED_APS - all running on a single
database. Writing a DATABASE_USING setting like the one you describe
would be extremely cumbersome and annoying.

So, we could hijack INSTALLED_APPS to represent the idea of database
deployment, using tuples/dictionaries rather than strings to define
how apps are deployed. However, this comes at the cost of backwards
compatibility for anyone iterating over INSTALLED_APPS.

Even if we were able to find an acceptable way to represent this
information, consider some of the other use cases for multi-db like
master/slave or sharding

Re: Multiple database support: Request for feedback and testing

[Paul McLanahan]

On Fri, Dec 4, 2009 at 5:51 PM, Yuri Baburov  wrote:
> #Isn't it django.contrib.auth? I'm not sure if collision with
> 'my.superstuff.auth' might happen or not.

It is, but the app registry mechanism in the
'django.db.models.loading' module only uses the "app label", which is
the part of the app module name after the last '.'. So, when using
django.db.models.get_model, you would pass it 'auth'. So if you named
your app 'my.superstuff.auth', the app used when calling
get_model('auth', 'user') might not be the one you'd expect.

That's my understanding anyway. I agree that it seems potentially fragile.

Paul

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

AFAICT adding a setting for the default manager just doesn't work because
then any models with custom managers either (a) would lose their custom
manager, or (b) wouldn't get the manager with using().

Cheers,
Tobias

On Fri, Dec 4, 2009 at 3:42 PM, Yuri Baburov  wrote:

> Hi all,
>
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
>
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.
>
> And provide with django few most usable enhanced managers replacements.
> This will solve both master/master, master/slave sharding and "this
> table is at DB1, that is at DB2" use case (how to name it in a single
> word?).
> There should also be such general solution for complex model save
> override. Maybe, Or Model.save calling such customizable method.
>
> Alex, you just don't get it seriously yet. Imagine the following:
> In your project, you use the plugin application A written by other
> people (and 25 other plugins B, C, D, ..., Z!).
> It defines model A, and 3 views, each one use model A, with some
> forms, templates, etc.
> How are you going to change A.save and managers while keeping the views
> working?
> Monkeypatch it? Make patched version for each app? Well, then please
> provide a method which will do this override for all models if called
> from a single place in project, probably, settings.
>
> In large projects, single point of override on per-project basis is
> required, not per-model.
>
> On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty 
> wrote:
> > On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan 
> wrote:
> >> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor 
> wrote:
> >>> We will not be adding a setting to pin an application/model to a
> >>> specific database.  We have already removed the Meta.using option.  If
> >>> you want to pin an application to a specific database it's fairly
> >>> trivial to add a manager and override the save() method to use a
> >>> specific DB.  Our goal is to design a good long term API for having a
> >>> flexible way to define exactly what DB any query goes to, and I
> >>> personally have no interest in seeing short term stopgap APIs
> >>> implemented.
> >>
> >> That's fine for apps under my control, but this leaves us without a
> >> way of dealing with 3rd party apps outside of an unfortunate amount of
> >> monkey patching. I certainly don't want the API to be "sort term" or
> >> "stopgap", but I do want this ability. I don't see any other way to
> >> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
> >> know and agree with the reasons that the Meta.using option was
> >> removed, but it wouldn't have solved these issues either. Adding a key
> >> to the DB configs may not be the right way, but I'm sure there is a
> >> good way to accomplish this.
> >
> > Agreed.  I was just writing basically the same reply.  And in our
> > defense, support for this functionality is the first thing one sees in
> > the "Requirements" section on the Multi DB support wiki page:
> >
> > http://code.djangoproject.com/wiki/MultipleDatabaseSupport#Requirements
> >
> > I see nothing "short term" or "stopgap" about this; rather, it solves
> > a specific and potentially common use case for multidb support in an
> > elegant, extensible way that also fixes other issues with the current
> > implementation (namely, the admin).
> >
> > That said, figuring out the right way to implement it is still up in
> > the air.  I proposed a separate setting; Paul proposed one integrated
> > with the DATABASES setting.  There are likely better ways to do it and
> > I'm not attached to any particular one, but the functionality is key.
> >
> > Cheers,
> > Tobias
> > --
> > Tobias McNulty
> > Caktus Consulting Group, LLC
> > P.O. Box 1454
> > Carrboro, NC 27510
> > (919) 951-0052
> > http://www.caktusgroup.com
> >
> > --
> >
> > You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> > To post to this group, send email to django-develop...@googlegroups.com.
> > To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> > For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
> >
> >
> >
>
>
>
> --
> Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
> MSN: bu...@live.com
>
> --
>
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscr...@googlegroups.com
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
>
>
>


-- 
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 145

Re: Multiple database support: Request for feedback and testing

[Yuri Baburov]

Hi Paul, Tobias,

Well, this is god.
Few more use cases that will appear on big django installations:
Sometimes, with master/slave replication (or legacy read-only
databases), users write to one connection, but read from (few) others.
Sometimes, with master/master replication, or different sharding
schemes, users write and read from random or key based database
connections.
So I attempted to modify this proposal to make this possible.

On Sat, Dec 5, 2009 at 3:31 AM, Paul McLanahan  wrote:
> On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty  wrote:
>> What would this look like?  I'm picturing another setting (bleh) that
>> maps apps and/or models to specific databases.  Name choices aside,
>> this might look something like:
>>
>> DATABASE_USING = {
>>     'django.contrib.auth': 'default',
>>     'myapp': 'database1',
>>     'myapp2': {
>>     'Model1': 'database2',
>>     'Model2': 'database3',
>>     }
>> }
>
> I like this. I think it could be made even more simple by using the
> "app_label.model_name" pattern in use by other settings (e.g.
> ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
> implementation easier using django.db.models.get_model and
> django.db.models.get_models.
>
> DATABASE_USING = {
>    'auth' : 'default',
#Isn't it django.contrib.auth? I'm not sure if collision with
'my.superstuff.auth' might happen or not.
>    'myapp' : 'database1',
>    'myapp2.model1' : 'database2',
>    'myapp2.model2' : 'database3',
 'myapp': ('database1', 'database2'), # random access
 'myapp3': {'read':('database1', 'database2'),
'write':my_get_app3_connection, 'syncdb':('default', 'database1',
'database2', 'database3')}, # custom function to get RW connection,
custom syncdb since .
 '*':my_get_other_connection, # for other 25 apps, use this
connection, not 'default'.
> }
Where my_get_*_connection function accept model name string, access
mode, and will return:
 for 'read' and 'write' mode, a database name or connection,
 for 'syncdb' mode, a list of database names.
>
> Any model not specifically mentioned could assume that it should use 
> "default".
Hope it isn't yet t complex to understand

This will also provide super-giper-mega-option:
DATABASE_USING = {'*': my_get_other_connection}
with these 3 modes.

-- 
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Yuri Baburov]

Hi James,

I just suggested a way it *could* work, not how it should work.

Are you objecting to my whole plan or only on Manager part of it?

On Sat, Dec 5, 2009 at 2:49 AM, James Bennett  wrote:
> On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov  wrote:
>> Like, calling current Manager a DefaultManager, and making "Manager =
>> load(settings.MODEL_MANAGER) or DefaultManager". Or
>> Manager.get_queryset calling customizable method. Any.
>
> If a suggestion like this is going to be implemented, I'd prefer it to
> be part of the database backend rather than a setting, because:
>
> 1. We already have a ton of DB-related settings, possibly more than we need.
Sorry, i don't count this an argument, if a question is "where there
is a single place to put project-wide configuration".
If we decided that settings.py is that storage -- let us be consistent.

> 2. Backends already support things like a custom Query class, and so
> this would be the logical place for it to go.
I will agree with you if you give me an example.

> 3. Encapsulating this as part of a DB backend most likely gives you a
> lot more freedom to build in all sorts of complex enterprisey stuff if
> you really want it.
Ehm Don't get it.
I.e, some of my tables are local sqlite database, some are remote
postgresql service, some are mysql on next server, some are in mssql
database accessed through django-pyodbc, and some tables are going to
be splitted across 16 bsddb files with custom backend, sharding based
on key hashes?
Where do I point which table should use what backend and which
connection do I use?

-- 
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Paul McLanahan]

On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty  wrote:
> What would this look like?  I'm picturing another setting (bleh) that
> maps apps and/or models to specific databases.  Name choices aside,
> this might look something like:
>
> DATABASE_USING = {
>     'django.contrib.auth': 'default',
>     'myapp': 'database1',
>     'myapp2': {
>     'Model1': 'database2',
>     'Model2': 'database3',
>     }
> }

I like this. I think it could be made even more simple by using the
"app_label.model_name" pattern in use by other settings (e.g.
ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
implementation easier using django.db.models.get_model and
django.db.models.get_models.

DATABASE_USING = {
'auth' : 'default',
'myapp' : 'database1',
'myapp2.model1' : 'database2',
'myapp2.model2' : 'database3',
}

Any model not specifically mentioned could assume that it should use "default".

Paul

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[James Bennett]

On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov  wrote:
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.

If a suggestion like this is going to be implemented, I'd prefer it to
be part of the database backend rather than a setting, because:

1. We already have a ton of DB-related settings, possibly more than we need.

2. Backends already support things like a custom Query class, and so
this would be the logical place for it to go.

3. Encapsulating this as part of a DB backend most likely gives you a
lot more freedom to build in all sorts of complex enterprisey stuff if
you really want it.


-- 
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Yuri Baburov]

Yes, and the similar function for syncdb can be written too!

On Sat, Dec 5, 2009 at 2:42 AM, Yuri Baburov  wrote:
> Hi all,
>
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
>
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.
>
> And provide with django few most usable enhanced managers replacements.
> This will solve both master/master, master/slave sharding and "this
> table is at DB1, that is at DB2" use case (how to name it in a single
> word?).
> There should also be such general solution for complex model save
> override. Maybe, Or Model.save calling such customizable method.
>
> Alex, you just don't get it seriously yet. Imagine the following:
> In your project, you use the plugin application A written by other
> people (and 25 other plugins B, C, D, ..., Z!).
> It defines model A, and 3 views, each one use model A, with some
> forms, templates, etc.
> How are you going to change A.save and managers while keeping the views 
> working?
> Monkeypatch it? Make patched version for each app? Well, then please
> provide a method which will do this override for all models if called
> from a single place in project, probably, settings.
>
> In large projects, single point of override on per-project basis is
> required, not per-model.
>
> On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty  wrote:
>> On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan  wrote:
>>> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor  wrote:
 We will not be adding a setting to pin an application/model to a
 specific database.  We have already removed the Meta.using option.  If
 you want to pin an application to a specific database it's fairly
 trivial to add a manager and override the save() method to use a
 specific DB.  Our goal is to design a good long term API for having a
 flexible way to define exactly what DB any query goes to, and I
 personally have no interest in seeing short term stopgap APIs
 implemented.
>>>
>>> That's fine for apps under my control, but this leaves us without a
>>> way of dealing with 3rd party apps outside of an unfortunate amount of
>>> monkey patching. I certainly don't want the API to be "sort term" or
>>> "stopgap", but I do want this ability. I don't see any other way to
>>> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
>>> know and agree with the reasons that the Meta.using option was
>>> removed, but it wouldn't have solved these issues either. Adding a key
>>> to the DB configs may not be the right way, but I'm sure there is a
>>> good way to accomplish this.
>>
>> Agreed.  I was just writing basically the same reply.  And in our
>> defense, support for this functionality is the first thing one sees in
>> the "Requirements" section on the Multi DB support wiki page:
>>
>> http://code.djangoproject.com/wiki/MultipleDatabaseSupport#Requirements
>>
>> I see nothing "short term" or "stopgap" about this; rather, it solves
>> a specific and potentially common use case for multidb support in an
>> elegant, extensible way that also fixes other issues with the current
>> implementation (namely, the admin).
>>
>> That said, figuring out the right way to implement it is still up in
>> the air.  I proposed a separate setting; Paul proposed one integrated
>> with the DATABASES setting.  There are likely better ways to do it and
>> I'm not attached to any particular one, but the functionality is key.
>>
>> Cheers,
>> Tobias
>> --
>> Tobias McNulty
>> Caktus Consulting Group, LLC
>> P.O. Box 1454
>> Carrboro, NC 27510
>> (919) 951-0052
>> http://www.caktusgroup.com
>>
>> --
>>
>> You received this message because you are subscribed to the Google Groups 
>> "Django developers" group.
>> To post to this group, send email to django-develop...@googlegroups.com.
>> To unsubscribe from this group, send email to 
>> django-developers+unsubscr...@googlegroups.com.
>> For more options, visit this group at 
>> http://groups.google.com/group/django-developers?hl=en.
>>
>>
>>
>
>
>
> --
> Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
> MSN: bu...@live.com
>



-- 
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Yuri Baburov]

Hi all,

Database -> Model.Manager -> Queryset.
Save -> Model -> Database

How about making a default Manager that's now "model.Manager" to be a
setting to override?
Like, calling current Manager a DefaultManager, and making "Manager =
load(settings.MODEL_MANAGER) or DefaultManager". Or
Manager.get_queryset calling customizable method. Any.

And provide with django few most usable enhanced managers replacements.
This will solve both master/master, master/slave sharding and "this
table is at DB1, that is at DB2" use case (how to name it in a single
word?).
There should also be such general solution for complex model save
override. Maybe, Or Model.save calling such customizable method.

Alex, you just don't get it seriously yet. Imagine the following:
In your project, you use the plugin application A written by other
people (and 25 other plugins B, C, D, ..., Z!).
It defines model A, and 3 views, each one use model A, with some
forms, templates, etc.
How are you going to change A.save and managers while keeping the views working?
Monkeypatch it? Make patched version for each app? Well, then please
provide a method which will do this override for all models if called
from a single place in project, probably, settings.

In large projects, single point of override on per-project basis is
required, not per-model.

On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty  wrote:
> On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan  wrote:
>> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor  wrote:
>>> We will not be adding a setting to pin an application/model to a
>>> specific database.  We have already removed the Meta.using option.  If
>>> you want to pin an application to a specific database it's fairly
>>> trivial to add a manager and override the save() method to use a
>>> specific DB.  Our goal is to design a good long term API for having a
>>> flexible way to define exactly what DB any query goes to, and I
>>> personally have no interest in seeing short term stopgap APIs
>>> implemented.
>>
>> That's fine for apps under my control, but this leaves us without a
>> way of dealing with 3rd party apps outside of an unfortunate amount of
>> monkey patching. I certainly don't want the API to be "sort term" or
>> "stopgap", but I do want this ability. I don't see any other way to
>> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
>> know and agree with the reasons that the Meta.using option was
>> removed, but it wouldn't have solved these issues either. Adding a key
>> to the DB configs may not be the right way, but I'm sure there is a
>> good way to accomplish this.
>
> Agreed.  I was just writing basically the same reply.  And in our
> defense, support for this functionality is the first thing one sees in
> the "Requirements" section on the Multi DB support wiki page:
>
> http://code.djangoproject.com/wiki/MultipleDatabaseSupport#Requirements
>
> I see nothing "short term" or "stopgap" about this; rather, it solves
> a specific and potentially common use case for multidb support in an
> elegant, extensible way that also fixes other issues with the current
> implementation (namely, the admin).
>
> That said, figuring out the right way to implement it is still up in
> the air.  I proposed a separate setting; Paul proposed one integrated
> with the DATABASES setting.  There are likely better ways to do it and
> I'm not attached to any particular one, but the functionality is key.
>
> Cheers,
> Tobias
> --
> Tobias McNulty
> Caktus Consulting Group, LLC
> P.O. Box 1454
> Carrboro, NC 27510
> (919) 951-0052
> http://www.caktusgroup.com
>
> --
>
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>
>



-- 
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan  wrote:
> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor  wrote:
>> We will not be adding a setting to pin an application/model to a
>> specific database.  We have already removed the Meta.using option.  If
>> you want to pin an application to a specific database it's fairly
>> trivial to add a manager and override the save() method to use a
>> specific DB.  Our goal is to design a good long term API for having a
>> flexible way to define exactly what DB any query goes to, and I
>> personally have no interest in seeing short term stopgap APIs
>> implemented.
>
> That's fine for apps under my control, but this leaves us without a
> way of dealing with 3rd party apps outside of an unfortunate amount of
> monkey patching. I certainly don't want the API to be "sort term" or
> "stopgap", but I do want this ability. I don't see any other way to
> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
> know and agree with the reasons that the Meta.using option was
> removed, but it wouldn't have solved these issues either. Adding a key
> to the DB configs may not be the right way, but I'm sure there is a
> good way to accomplish this.

Agreed.  I was just writing basically the same reply.  And in our
defense, support for this functionality is the first thing one sees in
the "Requirements" section on the Multi DB support wiki page:

http://code.djangoproject.com/wiki/MultipleDatabaseSupport#Requirements

I see nothing "short term" or "stopgap" about this; rather, it solves
a specific and potentially common use case for multidb support in an
elegant, extensible way that also fixes other issues with the current
implementation (namely, the admin).

That said, figuring out the right way to implement it is still up in
the air.  I proposed a separate setting; Paul proposed one integrated
with the DATABASES setting.  There are likely better ways to do it and
I'm not attached to any particular one, but the functionality is key.

Cheers,
Tobias
-- 
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
(919) 951-0052
http://www.caktusgroup.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Paul McLanahan]

On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor  wrote:
> We will not be adding a setting to pin an application/model to a
> specific database.  We have already removed the Meta.using option.  If
> you want to pin an application to a specific database it's fairly
> trivial to add a manager and override the save() method to use a
> specific DB.  Our goal is to design a good long term API for having a
> flexible way to define exactly what DB any query goes to, and I
> personally have no interest in seeing short term stopgap APIs
> implemented.

That's fine for apps under my control, but this leaves us without a
way of dealing with 3rd party apps outside of an unfortunate amount of
monkey patching. I certainly don't want the API to be "sort term" or
"stopgap", but I do want this ability. I don't see any other way to
accomplish a heterogeneous admin or partitioning of 3rd party apps. I
know and agree with the reasons that the Meta.using option was
removed, but it wouldn't have solved these issues either. Adding a key
to the DB configs may not be the right way, but I'm sure there is a
good way to accomplish this.

Thanks,

Paul

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Alex Gaynor]

On Fri, Dec 4, 2009 at 2:41 PM, Paul McLanahan  wrote:
> What will be the default action of manage.py syncdb (without the
> --database option)? Will it create all tables on all DBs, or just use
> "default"? The former would be useful for the simple replication use
> case, but wasteful for partitioning, and the latter could get tiresome
> unless there is a "--database all" option of which I'm unaware.
>
> I also don't see anywhere in the proposed docs where one could specify
> "app -> db" or "app.model -> db" mappings. Without some mechanism for
> such mapping, I'm afraid the partitioning case will become error prone
> and verbose, both with manage.py commands, and in app code. I'd hate
> to have to refactor all of my app's views if I moved some tables to a
> new server. Not having such mappings will also require a refactor of
> many of the views in existing distributed apps to become "multi-db
> enabled" if they don't already accept a QuerySet as an argument (isn't
> necessarily a bad thing, but could delay multi-db usefulness). Moving
> this problem to settings should (operative word) keep multi-db app
> related modification to a minimum, and thus not split all available
> apps into supporting and non-supporting categories. It could also
> greatly ease deployment if one didn't partition in dev, but did in
> staging and production. And this would go a long way toward fixing the
> Admin issue for the partitioning case, and for replication most will
> only need the admin on the default db anyway.
>
> The mappings could be as simple as a new key on the DATABASES dict:
>
>    DATABASES = {
>        'default': {
>            'BACKEND': 'django.db.backends.sqlite3',
>            'NAME': 'mydatabase',
>        },
>        'stuff': {
>            'BACKEND': 'django.db.backends.sqlite3',
>            'NAME': 'myotherdatabase',
>            'PINNED': ['myapp', 'someapp.model'],
>        },
>    }
>
> I thought I saw some discussion of this type of thing a while back.
> Was a mechanism for this delayed, removed, undocumented, or am I just
> missing something?
>
> Thanks,
>
> Paul
>
> --
>
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>
>

syncdb only syncs the default database by default.

We will not be adding a setting to pin an application/model to a
specific database.  We have already removed the Meta.using option.  If
you want to pin an application to a specific database it's fairly
trivial to add a manager and override the save() method to use a
specific DB.  Our goal is to design a good long term API for having a
flexible way to define exactly what DB any query goes to, and I
personally have no interest in seeing short term stopgap APIs
implemented.

Alex

-- 
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Paul McLanahan]

What will be the default action of manage.py syncdb (without the
--database option)? Will it create all tables on all DBs, or just use
"default"? The former would be useful for the simple replication use
case, but wasteful for partitioning, and the latter could get tiresome
unless there is a "--database all" option of which I'm unaware.

I also don't see anywhere in the proposed docs where one could specify
"app -> db" or "app.model -> db" mappings. Without some mechanism for
such mapping, I'm afraid the partitioning case will become error prone
and verbose, both with manage.py commands, and in app code. I'd hate
to have to refactor all of my app's views if I moved some tables to a
new server. Not having such mappings will also require a refactor of
many of the views in existing distributed apps to become "multi-db
enabled" if they don't already accept a QuerySet as an argument (isn't
necessarily a bad thing, but could delay multi-db usefulness). Moving
this problem to settings should (operative word) keep multi-db app
related modification to a minimum, and thus not split all available
apps into supporting and non-supporting categories. It could also
greatly ease deployment if one didn't partition in dev, but did in
staging and production. And this would go a long way toward fixing the
Admin issue for the partitioning case, and for replication most will
only need the admin on the default db anyway.

The mappings could be as simple as a new key on the DATABASES dict:

DATABASES = {
'default': {
'BACKEND': 'django.db.backends.sqlite3',
'NAME': 'mydatabase',
},
'stuff': {
'BACKEND': 'django.db.backends.sqlite3',
'NAME': 'myotherdatabase',
'PINNED': ['myapp', 'someapp.model'],
},
}

I thought I saw some discussion of this type of thing a while back.
Was a mechanism for this delayed, removed, undocumented, or am I just
missing something?

Thanks,

Paul

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

The very first "Requirement"/ideal feature on the wiki page reads:

"Different models/applications living on different databases, for
example a 'blog' application on db1 and a forum application on db2.
This should include the ability to assign a different database to an
existing application without modifying it, e.g. telling Django where
to keep the django.contrib.auth.User table. "

That sounds perfect - having a way to modify in what database a table
exists on a per-model or per-app basis, without making any changes to
the app itself (so I can choose to put reusable app A in database X
and reusable app B in database Y).

I don't see anything in the docs about if and how this type of
partitioning is supported.  From what I understand, it has been
proposed that this be implemented for the admin specifically through
ModelAdmin and/or at the admin.Site level, because in the current
iteration of the code there is no way to use the admin for anything
other than the default database.

To me, specifying the database in the ModelAdmin or admin.Site seems
arbitrary and potentially limiting: For any reusable app on the
market, depending on the value of a particular setting in your urls.py
or admin.py file is a Bad Idea.  What if the admin was instead fixed
by providing facilities for the more general case outlined above?

What would this look like?  I'm picturing another setting (bleh) that
maps apps and/or models to specific databases.  Name choices aside,
this might look something like:

DATABASE_USING = {
    'django.contrib.auth': 'default',
    'myapp': 'database1',
    'myapp2': {
    'Model1': 'database2',
    'Model2': 'database3',
    }
}

The admin could automatically take advantage of this setting to
determine what database to use for a given model.

Cheers,
Tobias

On Fri, Dec 4, 2009 at 10:47 AM, Russell Keith-Magee
 wrote:
>
> On Fri, Dec 4, 2009 at 11:40 PM, Tobias McNulty  
> wrote:
> > Is there a page where one can find a quick summary of the proposed API?
> >
> > I have some concerns about implementing partitioning through the admin, but
> > I expect there's something I'm missing.
> >
> > For those who haven't been, following the conversation closely it'd be nice
> > to have a quick way to come up to speed.
>
> The proposed API is intentionally simple - QuerySets gain a .using()
> modifier that lets you direct a query at a specific database; save()
> gains a using argument to let you save to a specific database. There
> are some configuration changes and to the get_db_prep_* method on
> Field to support multiple database setups, but that's about it. Most
> of the changes have been in the internals to make the two external
> APIs possible.
>
> As noted a couple of messages back, support in the admin was an
> oversight. The existing admin will work for the default database, but
> will require modifications to support other databases.
>
> For more details, the links I gave in the original call for testing
> are probably your best bet:
>
> http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt
>
> http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt
>
> Yours,
> Russ Magee %-)
>
> --
>
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.
>
>



--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
(919) 951-0052
http://www.caktusgroup.com

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Fri, Dec 4, 2009 at 11:40 PM, Tobias McNulty  wrote:
> Is there a page where one can find a quick summary of the proposed API?
>
> I have some concerns about implementing partitioning through the admin, but
> I expect there's something I'm missing.
>
> For those who haven't been, following the conversation closely it'd be nice
> to have a quick way to come up to speed.

The proposed API is intentionally simple - QuerySets gain a .using()
modifier that lets you direct a query at a specific database; save()
gains a using argument to let you save to a specific database. There
are some configuration changes and to the get_db_prep_* method on
Field to support multiple database setups, but that's about it. Most
of the changes have been in the internals to make the two external
APIs possible.

As noted a couple of messages back, support in the admin was an
oversight. The existing admin will work for the default database, but
will require modifications to support other databases.

For more details, the links I gave in the original call for testing
are probably your best bet:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Tobias McNulty]

Is there a page where one can find a quick summary of the proposed API?

I have some concerns about implementing partitioning through the admin, but
I expect there's something I'm missing.

For those who haven't been, following the conversation closely it'd be nice
to have a quick way to come up to speed.

Sent from a mobile phone, please excuse any typos.

On Dec 4, 2009 9:30 AM, "Russell Keith-Magee" 
wrote:

On Fri, Dec 4, 2009 at 10:18 PM, Nan  wrote: > >> 1)
Ignore the problem. Admin w...
This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.

Yours,
Russ Magee %-)

-- You received this message because you are subscribed to the Google Groups
"Django developers" g...

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Fri, Dec 4, 2009 at 10:18 PM, Nan  wrote:
>
>> 1) Ignore the problem. Admin works on the default database, but
>> nowhere else. This is certainly less than ideal, but it would be
>> sufficient for master/slave setups.
>>
>> 2) Use a separate admin deployment for each database. We add a 'using'
>> argument to admin.Site(), and append .using() the queries that the
>> admin site issues.
>>
>> 3) Modify the admin list views so that they take a ?using=db GET
>> argument; also add a pulldown to make it easy to switch to a different
>> database.
>>
>> (2) should be a fairly minor change; (3) would be a little more
>> effort, but shouldn't be impossible. There is also the need for some
>> mechanics to check whether a table is actually present on a database -
>> just because auth is in INSTALLED_APPS doesn't mean it's been
>> synchronized to a particular database.
>>
>> I'm open to any other suggestions - and for any offers to help out :-)
>
> Just thinking of one more option for this -- what about specifying the
> DB on the ModelAdmin level rather than the admin.Site level?

This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Nan]

> 1) Ignore the problem. Admin works on the default database, but
> nowhere else. This is certainly less than ideal, but it would be
> sufficient for master/slave setups.
>
> 2) Use a separate admin deployment for each database. We add a 'using'
> argument to admin.Site(), and append .using() the queries that the
> admin site issues.
>
> 3) Modify the admin list views so that they take a ?using=db GET
> argument; also add a pulldown to make it easy to switch to a different
> database.
>
> (2) should be a fairly minor change; (3) would be a little more
> effort, but shouldn't be impossible. There is also the need for some
> mechanics to check whether a table is actually present on a database -
> just because auth is in INSTALLED_APPS doesn't mean it's been
> synchronized to a particular database.
>
> I'm open to any other suggestions - and for any offers to help out :-)

Just thinking of one more option for this -- what about specifying the
DB on the ModelAdmin level rather than the admin.Site level?

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Doug Blank]

On Fri, Dec 4, 2009 at 6:31 AM, Russell Keith-Magee
 wrote:

[snip]

>  1) Do nothing special. This is the currently implement behaviour.

As the default, this seems to be the intended behavior only in rare
cases (overwriting a different object from the original) and will
indeed bite.

There is another option: leave in the hands of the user (as you say)
but make them explicitly say which behavior they want of the
following:

>  2) Set the primary key of the object to None so that the save on the
> new database is guaranteed to be an insertion, rather than a possible
> overwrite of an object with the same pk value on the second database.
>
>  3) Transform the save into a 'force insert' if the database changes.
> This will raise errors if the pk is already in use on
>
>  4) Use the PK on the old database and issue a deletion before saving
> to the new database.

If they didn't explicitly indicate which of these to do, then raise an error.

Just a thought.

-Doug

[snip]

> I'm open to any other suggestions - and for any offers to help out :-)
>
> Russ %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Fri, Dec 4, 2009 at 7:31 PM, Russell Keith-Magee
 wrote:
> IMHO, it's best to leave entirely in the hands of the end user. By
> that reasoning, The current behaviour (1) is actually the right
> solution, along with some documentation explaining the problem and
> some possible solutions - essentially, the manual interpretations of
> (2) and (3).

I've just added some docs to the github branch to cover this situation:

http://github.com/alex/django/commit/9c162e50c81883e214b60f749de81b79ee10ace6

Russ %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Fri, Dec 4, 2009 at 4:38 PM, Simon Willison  wrote:
> On Dec 3, 4:10 pm, Russell Keith-Magee  wrote:
>> Alex Gaynor's GSoC project to add multiple database support to Django
>> is on the final straight. The only piece of the puzzle that is left is
>> updating contrib.gis - but this hopefully won't require any major
>> changes outside of the gis tree itself.
>>
>> Therefore, I'd like to call for feedback and testing of the branch.
>
> I tried this out yesterday - it's excellent. The combination of the
> new DATABASES setting and QuerySet.using() turns out to be exactly
> enough to get all sorts of interesting work done. It neatly fulfils
> the three requirements set out in the original ticket (replication /
> sharding / hosting different applications on different databases).
>
> A couple of observations. Firstly, there's a gotcha involving saving
> existing objects to a new database:
>
> e1 = Entry.objects.using('db1').create(title = "A blog post",
> body="...")
> e2 = Entry.objects.using('db1').create(title = "Another blog post",
> body="...")
> e3 = Entry.objects.using('db2').create(title = "Another blog post",
> body="...")
> e4 = Entry.objects.using('db2').create(title = "Another blog post",
> body="...")
>
> We now have two blog posts in each database. But, if I do the
> following:
>
> e1.save(using = 'db2')
>
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.
>
> This behaviour makes sense if you think about it for more than a
> moment, but still probably counts as a gotcha. Worth mentioning in the
> documentation at least.
>
> I could see this pattern biting people since that's the obvious way to
> copy an object from one database to another.

Agreed that this could bite people fairly easily.

I'm not sure we can really do much to avoid the problem, though. We
have a few options, but none of them are really appropriate for all
circumstances.

A time of object save, we know the database that the object is
currently on (using _state), and we know if the user has requested a
save on a different database (by comparing with the using argument).
Based on this detail, we could:

 1) Do nothing special. This is the currently implement behaviour.

 2) Set the primary key of the object to None so that the save on the
new database is guaranteed to be an insertion, rather than a possible
overwrite of an object with the same pk value on the second database.

 3) Transform the save into a 'force insert' if the database changes.
This will raise errors if the pk is already in use on

 4) Use the PK on the old database and issue a deletion before saving
to the new database.

 5) Do 2 and 4.

 6) Do 3 and 4

The problem is the conflict between the most appropriate behavior for
different use cases. In the case you describe (copying an object from
one database to another), (5) probably makes the most sense; however,
from the point of view of a master/slave setup, (1) is the best
behaviour.

In fact (4) by makes me a little nervous - If we start modifying
save() to make it copy objects, there will be surprise when objects
related to the object being saved aren't transferred as well. Deleting
data from the old database sounds like a recipe for some really nasty
accidental data loss.

IMHO, it's best to leave entirely in the hands of the end user. By
that reasoning, The current behaviour (1) is actually the right
solution, along with some documentation explaining the problem and
some possible solutions - essentially, the manual interpretations of
(2) and (3).

> Secondly, a question: Is there an idea for how this might be made to
> work with the admin interface? I was asked this question at DJUGL last
> night and wasn't sure of the answer.

...and this is why we put code out for review. You win the internets
for pointing out the obvious omission. For all the spelunking in
django.db internals, the potential effect on admin hadn't occurred to
me.

I can think of a couple of possible approaches to this.

1) Ignore the problem. Admin works on the default database, but
nowhere else. This is certainly less than ideal, but it would be
sufficient for master/slave setups.

2) Use a separate admin deployment for each database. We add a 'using'
argument to admin.Site(), and append .using() the queries that the
admin site issues.

3) Modify the admin list views so that they take a ?using=db GET
argument; also add a pulldown to make it easy to switch to a different
database.

(2) should be a fairly minor change; (3) would be a little more
effort, but shouldn't be impossible. There is also the need for some
mechanics to c

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 4, 8:38 am, Simon Willison  wrote:
> We now have two blog posts in each database. But, if I do the
> following:
>
> e1.save(using = 'db2')
>
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.

I meant to say - the fix is clearly to call save() with the
force_insert=True argument, but it's easy to miss that. Worth
explicitly highlighting in the multi-db docs I think.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Simon Willison]

On Dec 3, 4:10 pm, Russell Keith-Magee  wrote:
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
>
> Therefore, I'd like to call for feedback and testing of the branch.

I tried this out yesterday - it's excellent. The combination of the
new DATABASES setting and QuerySet.using() turns out to be exactly
enough to get all sorts of interesting work done. It neatly fulfils
the three requirements set out in the original ticket (replication /
sharding / hosting different applications on different databases).

A couple of observations. Firstly, there's a gotcha involving saving
existing objects to a new database:

e1 = Entry.objects.using('db1').create(title = "A blog post",
body="...")
e2 = Entry.objects.using('db1').create(title = "Another blog post",
body="...")
e3 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")
e4 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")

We now have two blog posts in each database. But, if I do the
following:

e1.save(using = 'db2')

It has the effect of replacing e3 with the values from e2, since the
primary keys collide with each other.

This behaviour makes sense if you think about it for more than a
moment, but still probably counts as a gotcha. Worth mentioning in the
documentation at least.

I could see this pattern biting people since that's the obvious way to
copy an object from one database to another.

Secondly, a question: Is there an idea for how this might be made to
work with the admin interface? I was asked this question at DJUGL last
night and wasn't sure of the answer.

Fantastic work on this, very excited to see it merged to trunk.

Cheers,

Simon

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

On Fri, Dec 4, 2009 at 12:20 AM, Waldemar Kornewald
 wrote:
> Hi Russell,
> I just noticed a simple documentation bug. See the comment at the
> bottom of this page:
> http://github.com/alex/django/commit/aabfee1571d378dd3b7550573e900850d13e1b9b

You are correct. Thanks for that - I've made the change on github.

> Also, I hope you won't define an API for overriding the query class in
> the multi-db branch. We'll most likely have to define our own API in
> the non-relational branch.

We haven't modified the API for overriding the query class beyond the
techniques that already existed to support GIS (i.e., passing a Query
class into the QuerySet itself).

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support: Request for feedback and testing

[Waldemar Kornewald]

Hi Russell,
I just noticed a simple documentation bug. See the comment at the
bottom of this page:
http://github.com/alex/django/commit/aabfee1571d378dd3b7550573e900850d13e1b9b

Also, I hope you won't define an API for overriding the query class in
the multi-db branch. We'll most likely have to define our own API in
the non-relational branch.

Bye,
Waldemar Kornewald

On Thu, Dec 3, 2009 at 5:10 PM, Russell Keith-Magee
 wrote:
> Hi all,
>
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
>
> Therefore, I'd like to call for feedback and testing of the branch.
>
> Code is available on github:
>
> http://github.com/alex/django/tree/multiple-db
>
> It is also available on Django's own subversion repository in the
> soc2009/multidb branch:
>
> svn co http://code.djangoproject.com/svn/django/branches/soc2009/multidb
>
> This is a very big change to the query engine, but unless your code
> relies on the internals of sql.Query, you *should* be able to switch
> to using the branch without making any changes to your code. The
> release notes give the details of the changes that you need to be
> aware of:
>
> http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt
>
> If you want to run Django's test suite, you will need to make some
> modifications to your test settings file. If you don't make these
> changes, you will get failures in the multiple_databases tests. See
> the relevant section in the contribution docs for details on the
> changes you need to make:
>
> http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/internals/contributing.txt
>
> To the best of our knowledge, the full test suite is currently passing
> for all databases (except for those known pre-existing failures under
> MySQL InnoDB and Oracle). Any discoveries to the contrary will be
> gratefully received.
>
> If you actually want to try using multiple databases, basic usage
> instructions are here:
>
> http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt
>
> What should you expect to see? Well, a while back, Malcolm described
> Django's query framework as having the plumbing, but not the porcelain
> for multiple database support. At the risk of stretching the metaphor,
> this branch adds the porcelain, but not the polished oak seat to go on
> top. The settings and API to define multiple databases and to direct
> queries to a specific database have all been added by this branch.
> However, there isn't any explicit API level support for the common use
> cases for multiple databases like sharding or master/slave setups. I
> see adding these wrappers as tasks for the broader community, possibly
> with a view to inclusion in trunk in v1.3 or later.
>
> So - feedback welcome. It's a big patch, so the more testing we can
> get pre-merge, the better.
>
> Yours,
> Russ Magee %-)
>
> --
>
> You received this message because you are subscribed to the Google Groups 
> "Django developers" group.
> To post to this group, send email to django-develop...@googlegroups.com.
> To unsubscribe from this group, send email to 
> django-developers+unsubscr...@googlegroups.com.
> For more options, visit this group at 
> http://groups.google.com/group/django-developers?hl=en.

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Multiple database support: Request for feedback and testing

[Russell Keith-Magee]

Hi all,

Alex Gaynor's GSoC project to add multiple database support to Django
is on the final straight. The only piece of the puzzle that is left is
updating contrib.gis - but this hopefully won't require any major
changes outside of the gis tree itself.

Therefore, I'd like to call for feedback and testing of the branch.

Code is available on github:

http://github.com/alex/django/tree/multiple-db

It is also available on Django's own subversion repository in the
soc2009/multidb branch:

svn co http://code.djangoproject.com/svn/django/branches/soc2009/multidb

This is a very big change to the query engine, but unless your code
relies on the internals of sql.Query, you *should* be able to switch
to using the branch without making any changes to your code. The
release notes give the details of the changes that you need to be
aware of:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt

If you want to run Django's test suite, you will need to make some
modifications to your test settings file. If you don't make these
changes, you will get failures in the multiple_databases tests. See
the relevant section in the contribution docs for details on the
changes you need to make:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/internals/contributing.txt

To the best of our knowledge, the full test suite is currently passing
for all databases (except for those known pre-existing failures under
MySQL InnoDB and Oracle). Any discoveries to the contrary will be
gratefully received.

If you actually want to try using multiple databases, basic usage
instructions are here:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt

What should you expect to see? Well, a while back, Malcolm described
Django's query framework as having the plumbing, but not the porcelain
for multiple database support. At the risk of stretching the metaphor,
this branch adds the porcelain, but not the polished oak seat to go on
top. The settings and API to define multiple databases and to direct
queries to a specific database have all been added by this branch.
However, there isn't any explicit API level support for the common use
cases for multiple databases like sharding or master/slave setups. I
see adding these wrappers as tasks for the broader community, possibly
with a view to inclusion in trunk in v1.3 or later.

So - feedback welcome. It's a big patch, so the more testing we can
get pre-merge, the better.

Yours,
Russ Magee %-)

--

You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to 
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en.

Re: Multiple database support

[Jan Oberst]

I've been doing a little reading on multi-db code an wiki. You've
basically been tackling problem #3 (different data types and engines)
- which I didn't care about at all. That's good, I guess.

The way I handle database connections is just by having a connection
pool of different connection objects alive at the same time and create
new cursors on the connections I need. Since I have only implemented
the most simple SELECT FROM WHERE for one and many rows I haven't
worried too much about commit and rollback and stuff like that. So I
don't get all of your code and why you need to use threadlocals and
stuff like that.

The basic thing I do when a Django Model is sharded to different
server is this:

1. I write a get_shards classfunction for every model that does some
logic and returns a list of one or more shard objects that have a link
to this shard's connection.
2. Ask the model class on which shards it is, this returns a list of
shard objects.
3. For each shard object I get a new database cursor from the
connection which lives in a seperate connection object for every shard
(I'm not perfectly sure if this is thread safe)
4. For each of those cursors I repeat the query Django wanted to run.
Then I try to stich the responses together the best I can.

You guys obviously know your code better than me. Should I start re-
writing my code (necessary after queryset-refactoring) based on your
patch?

Jan

On Jul 8, 4:04 pm, "Ben Ford" <[EMAIL PROTECTED]> wrote:
> Hi Jan,
>
> It sounds like you've made great progress. We have an informal trac and hg
> repo set up at trac and hg dot woe-beti.de respectively. you're more than
> welcome to add your documentation there! Let me know if you want an hg repo
> tp play with too and I'll sort it out for you.
>
> Cheers,
> Ben
>
> 2008/7/8 Jan Oberst <[EMAIL PROTECTED]>:
>
>
>
>
>
> > Hi guys,
>
> > I've been heavily swamped with work for college, so I missed this
> > thead and the few others on multiple databases. Sorry.
>
> > I have implemented a proof-of-concept database scaling solution for
> > Django. It tackles all kind of scaling issues I have seen in Django.
> > It's purpose is mainly to find out if we could scale up Django at all.
> > I didn't worry too much about syntax and the way it's supposed to
> > integrate into Django - I just hacked away in Django code to make it
> > work the fastest possible way I could think of.
>
> > The solution covers the largest part of Simon's #2 problem. I added a
> > few attributes and config parameters to the ORM so you can decide
> > which models are hosted on which server. One model can be hosted on 20
> > servers with the actual location depending on a foreign key value.
>
> > We're using it to store data for different groups on different servers
> > for a more horizontal scaling. For example if a photo got a ForeignKey
> > to group A it will be routed to server 15 because of some logic.
>
> > You can also route objects 1-1000 to server 1 and 1001-2000 to server
> > 2.
>
> > I have also added database denormalization, caching foreign key
> > querysets to the DB, bulk prefetching, in-model privacy checks and a
> > few other things.
>
> > A large percentage of the stuff probably isn't suitable for Django-
> > trunk. Most of it tackles quite specific and hard scaling issues, but
> > I guess there's a way to build it more modular and make it work for
> > more people. After all I'm new to Django-developers and also to
> > opening up my work.
>
> > If some of you are interested in the code and would benefit from it I
> > would be more than happy to share.
>
> > Just posting a big pile of code probably won't help you too much, so I
> > thought I'd write a few lines documentation about each part and post
> > them here. Does that sound reasonable?
>
> > Jan
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ben Ford]

Hi Jan,

It sounds like you've made great progress. We have an informal trac and hg
repo set up at trac and hg dot woe-beti.de respectively. you're more than
welcome to add your documentation there! Let me know if you want an hg repo
tp play with too and I'll sort it out for you.

Cheers,
Ben

2008/7/8 Jan Oberst <[EMAIL PROTECTED]>:

>
> Hi guys,
>
> I've been heavily swamped with work for college, so I missed this
> thead and the few others on multiple databases. Sorry.
>
> I have implemented a proof-of-concept database scaling solution for
> Django. It tackles all kind of scaling issues I have seen in Django.
> It's purpose is mainly to find out if we could scale up Django at all.
> I didn't worry too much about syntax and the way it's supposed to
> integrate into Django - I just hacked away in Django code to make it
> work the fastest possible way I could think of.
>
>
> The solution covers the largest part of Simon's #2 problem. I added a
> few attributes and config parameters to the ORM so you can decide
> which models are hosted on which server. One model can be hosted on 20
> servers with the actual location depending on a foreign key value.
>
> We're using it to store data for different groups on different servers
> for a more horizontal scaling. For example if a photo got a ForeignKey
> to group A it will be routed to server 15 because of some logic.
>
> You can also route objects 1-1000 to server 1 and 1001-2000 to server
> 2.
>
>
> I have also added database denormalization, caching foreign key
> querysets to the DB, bulk prefetching, in-model privacy checks and a
> few other things.
>
> A large percentage of the stuff probably isn't suitable for Django-
> trunk. Most of it tackles quite specific and hard scaling issues, but
> I guess there's a way to build it more modular and make it work for
> more people. After all I'm new to Django-developers and also to
> opening up my work.
>
> If some of you are interested in the code and would benefit from it I
> would be more than happy to share.
>
> Just posting a big pile of code probably won't help you too much, so I
> thought I'd write a few lines documentation about each part and post
> them here. Does that sound reasonable?
>
> Jan
>
> On May 22, 4:59 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> > I have to admit I'm slightly worried about the multi-database
> > proposal, because at the moment it doesn't seem to solve either of the
> > multi-db problems I'm concerned about.
> >
> > The proposal at the moment deals with having different models live in
> > different databases - for example, the Forum application lives on DB1
> > while the Blog application lives on DB2.
> >
> > I can see how this could be useful, but the two database problems that
> > keep me up at night are the following:
> >
> > 1. Replication - being able to send all of my writes to one master
> > machine but spread all of my reads over several slave machines.
> > Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
> > problem neatly without modification to Django core - it's just an
> > alternative DB backend which demonstrates that doing this isn't
> > particularly hard:http://softwaremaniacs.org/soft/mysql_cluster/en/
> >
> > 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> > User entries 1001-2000 live on DB2 and so on.
> >
> > I'd love Django to have built-in abilities to solve #1 - it's a really
> > important first-step onscalingup to multiple databases, and it's
> > also massively easier than any other part of the multi-db problem.
> >
> > I wouldn't expect a magic solution to #2 because it's so highly
> > dependent on the application that is being built, but at the same time
> > it would be nice to see a multi-db solution at least take this in to
> > account (maybe just by providing an easy tool to direct an ORM request
> > to a specific server based on some arbitrary logic).
> >
> > I may have misunderstood the proposal, but I think it's vital that the
> > above two use cases are considered. Even if they can't be solved
> > outright, providing tools that custom solutions to these cases can be
> > built with should be a priority for multi-db support.
> >
> > Cheers,
> >
> > Simon
> >
>


-- 
Regards,
Ben Ford
[EMAIL PROTECTED]
+447792598685

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Jan Oberst]

Hi guys,

I've been heavily swamped with work for college, so I missed this
thead and the few others on multiple databases. Sorry.

I have implemented a proof-of-concept database scaling solution for
Django. It tackles all kind of scaling issues I have seen in Django.
It's purpose is mainly to find out if we could scale up Django at all.
I didn't worry too much about syntax and the way it's supposed to
integrate into Django - I just hacked away in Django code to make it
work the fastest possible way I could think of.


The solution covers the largest part of Simon's #2 problem. I added a
few attributes and config parameters to the ORM so you can decide
which models are hosted on which server. One model can be hosted on 20
servers with the actual location depending on a foreign key value.

We're using it to store data for different groups on different servers
for a more horizontal scaling. For example if a photo got a ForeignKey
to group A it will be routed to server 15 because of some logic.

You can also route objects 1-1000 to server 1 and 1001-2000 to server
2.


I have also added database denormalization, caching foreign key
querysets to the DB, bulk prefetching, in-model privacy checks and a
few other things.

A large percentage of the stuff probably isn't suitable for Django-
trunk. Most of it tackles quite specific and hard scaling issues, but
I guess there's a way to build it more modular and make it work for
more people. After all I'm new to Django-developers and also to
opening up my work.

If some of you are interested in the code and would benefit from it I
would be more than happy to share.

Just posting a big pile of code probably won't help you too much, so I
thought I'd write a few lines documentation about each part and post
them here. Does that sound reasonable?

Jan

On May 22, 4:59 pm, Simon Willison <[EMAIL PROTECTED]> wrote:
> I have to admit I'm slightly worried about the multi-database
> proposal, because at the moment it doesn't seem to solve either of the
> multi-db problems I'm concerned about.
>
> The proposal at the moment deals with having different models live in
> different databases - for example, the Forum application lives on DB1
> while the Blog application lives on DB2.
>
> I can see how this could be useful, but the two database problems that
> keep me up at night are the following:
>
> 1. Replication - being able to send all of my writes to one master
> machine but spread all of my reads over several slave machines.
> Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
> problem neatly without modification to Django core - it's just an
> alternative DB backend which demonstrates that doing this isn't
> particularly hard:http://softwaremaniacs.org/soft/mysql_cluster/en/
>
> 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> User entries 1001-2000 live on DB2 and so on.
>
> I'd love Django to have built-in abilities to solve #1 - it's a really
> important first-step onscalingup to multiple databases, and it's
> also massively easier than any other part of the multi-db problem.
>
> I wouldn't expect a magic solution to #2 because it's so highly
> dependent on the application that is being built, but at the same time
> it would be nice to see a multi-db solution at least take this in to
> account (maybe just by providing an easy tool to direct an ORM request
> to a specific server based on some arbitrary logic).
>
> I may have misunderstood the proposal, but I think it's vital that the
> above two use cases are considered. Even if they can't be solved
> outright, providing tools that custom solutions to these cases can be
> built with should be a priority for multi-db support.
>
> Cheers,
>
> Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[David Cramer]

I suppose I'll chime in here since we actually wrote master/slave
replication code on Curse.

Our approach:

- read_cursor and write_cursor exist. write_cursor is what cursor
would point ot.
- get queries all use the read cursor
- saves all use the write cursor
- we had a list of database connections, which stored the same
settings, just in a tuple format
- reading I believe used something like itertools.cycle but I can't
honestly say without looking at the code

Beyond this, the database itself should handle writing the objects to
the slaves. Django shouldn't even bother.

In regards to multiple databases in general. it is my feeling that
even if it is not good practice, Django _NEEDS_ to support a model
being attached to a database other than the default. So if you have
mydjango_blogs, and mydjango_forums databases, my Forum model always
goes to the write db when it queries, and same for blogs. I myself
like a Meta solution to this as it makes sense.

In MySQL as well, you can optimize things, so that if they use the
same connection, you can just query on that database. It's select X
from mydatabase.mytable. I'm not sure if something similar exists in
other database engines.

On Jun 16, 9:05 pm, mengel <[EMAIL PROTECTED]> wrote:
> On May 22, 9:59 am, Simon Willison <[EMAIL PROTECTED]> wrote:
>
> > 1. Replication - being able to send all of my writes to one master
> > machine but spread all of my reads over several slave machines.
> > 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> > User entries 1001-2000 live on DB2 and so on.
>
> It seems to me this isn't beyond doing in the current setup; but I'm
> not sure I understand
> the underlying mechanism well enough.   For case 1, you need an object
> class that
> creates two (or more) (apparently identical) Models.model classes, one
> attached to each database, based on the field types declared as class
> variables:
>   * on searches, it picks one of the model classes to search
>   * on saves, saves the same data to each  object class in turn
>
> For case 2, it's very similar, except you need to run the query on all
> sides (unless
> you can tell it should only go to one) building a chained query-set
> union type to hold
> the result, and for saves pick the right model  to save to  based on
> the condition.
>
> In each case, the underlying models have to be tied to the right
> databases, but this can
> be done using the mechanism in the proposal so far..
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[mengel]

On May 22, 9:59 am, Simon Willison <[EMAIL PROTECTED]> wrote:

> 1. Replication - being able to send all of my writes to one master
> machine but spread all of my reads over several slave machines.


> 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> User entries 1001-2000 live on DB2 and so on.

It seems to me this isn't beyond doing in the current setup; but I'm
not sure I understand
the underlying mechanism well enough.   For case 1, you need an object
class that
creates two (or more) (apparently identical) Models.model classes, one
attached to each database, based on the field types declared as class
variables:
  * on searches, it picks one of the model classes to search
  * on saves, saves the same data to each  object class in turn

For case 2, it's very similar, except you need to run the query on all
sides (unless
you can tell it should only go to one) building a chained query-set
union type to hold
the result, and for saves pick the right model  to save to  based on
the condition.

In each case, the underlying models have to be tied to the right
databases, but this can
be done using the mechanism in the proposal so far..

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Daryl Spitzer]

Another couple weeks have slipped by and I continue to be crazy-busy.
(But each week I'm busy for a different reason--so I continue to be
foolishly optimistic that I'll soon get a week with some free time.)

Anyway, I don't have time to read this thread through with the care it
deserves, but I thought I shouldn't let that stop me from finally
writing a description of the API I proposed at the PyCon sprint.

Each app would have a databases.py file that contains classes used to
define databases connections (in the same manner as classes are used
to define models).  Here's an example:



from django.db import connections

class LegacyDatabase(connections.DatabaseConnection):
   engine = 'sqlite3'
   name = '/foo/bar/legacy_db.sqlite3'



(And the other DATABASE_* settings (from settings.py) could certainly
be defined as attributes of a DatabaseConnection class.)

JUST FOR TESTING, I propose we allow a database connection to be
specified in a model with a Meta attribute, like this:



from django.db import models
from legacy.databases import LegacyDatabase

class LegacyStuff(models.Model):
...

class Meta:
db_connection = LegacyDatabase



Jacob expressed his extreme distaste for this at PyCon--for good
reason.  (We don't want to encourage coupling models to databases.)
But just so we can get a working patch and start testing, I propose we
go with this for now.

Adrian suggested we allow the specification of database connections
per-app using the new app() function being proposed for settings.py.
I haven't seen a description of this since PyCon, but I think it would
look something like:

app(name='legacy', db_connection='LegacyDatabase')

(I'm sure I'm leaving several important arguments out of this example.)

Perhaps one could implement sharding by defining multiple
DatabaseConnection classes in a databases.py file (we could support
these files at the project level in addition to the app level) and
putting them in a list.  Then one could write a function to return the
appropriate database to use and specify that callable in the argument
to the app function (or perhaps as an argument to the url function in
urls.py).

I haven't given any thought to replication.  Perhaps someone who needs
this could think about whether this proposal could somehow make
supporting replication easier (or if it might get in the way), or if
it's simply orthogonal to this.

--
Daryl


On Wed, May 28, 2008 at 12:25 AM, koenb <[EMAIL PROTECTED]> wrote:
>
> On 22 mei, 18:28, "Ben Ford" <[EMAIL PROTECTED]> wrote:
>> Hi all,
>>
>> I've now re-applied Daryls patch (which was against qsrf) to a clone of
>> django trunk in a mercurial repo. It's available 
>> athttp://hg.woe-beti.deandthere's a trac set up for it 
>> athttp://trac.woe-beti.de. Feel free to make use of both of these. Although
>> I've disabled to ability to create tickets perhaps the wiki might be a good
>> place to discuss the API? Anyone can clone from the hg repo, give me a shout
>> if you would like push access and I'll sort it out.
>>
>> Cheers,
>> Ben
>>
>> --
>> Regards,
>> Ben Ford
>> [EMAIL PROTECTED]
>> +447792598685
>
> I have been adding some code to Ben's mercurial repo on [http://hg.woe-
> beti.de], see also [http://trac.woe-beti.de].
>
> What has been realised (more or less):
> - connection and model registration
> - validation that related objects use the same connection
> - database_engine specific SQL generation (needs more checking)
> - some management commands accept connection parameter, others can
> generate output for multiple connections
> - syncdb can sync different connections
> - transaction management over connections
> - some tests (needs a lot more work)
>
> This means point 3 of the discussion at [http://trac.woe-beti.de/wiki/
> Discuss] is somewhat working, but definitely needs a lot more testing.
>
> I do need some help with creating tests for all this though.
> I have not figured out yet how to create tests that check that the
> right SQL is being generated for the backend used. (Nor how to test
> the right database was touched by an action, this is quite obvious
> manually, but I do not know how to automate this.)
>
> I put some ideas on using multiple databases for copying (backup or
> migration) of objects (point 4 of the discussion note) in [http://
> trac.woe-beti.de/wiki/APIModelDuplication].
>
> Please comment, add, shoot etc. Any help will be much appreciated.
>
> Koen
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[koenb]

On 22 mei, 18:28, "Ben Ford" <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I've now re-applied Daryls patch (which was against qsrf) to a clone of
> django trunk in a mercurial repo. It's available 
> athttp://hg.woe-beti.deandthere's a trac set up for it 
> athttp://trac.woe-beti.de. Feel free to make use of both of these. Although
> I've disabled to ability to create tickets perhaps the wiki might be a good
> place to discuss the API? Anyone can clone from the hg repo, give me a shout
> if you would like push access and I'll sort it out.
>
> Cheers,
> Ben
>
> --
> Regards,
> Ben Ford
> [EMAIL PROTECTED]
> +447792598685

I have been adding some code to Ben's mercurial repo on [http://hg.woe-
beti.de], see also [http://trac.woe-beti.de].

What has been realised (more or less):
- connection and model registration
- validation that related objects use the same connection
- database_engine specific SQL generation (needs more checking)
- some management commands accept connection parameter, others can
generate output for multiple connections
- syncdb can sync different connections
- transaction management over connections
- some tests (needs a lot more work)

This means point 3 of the discussion at [http://trac.woe-beti.de/wiki/
Discuss] is somewhat working, but definitely needs a lot more testing.

I do need some help with creating tests for all this though.
I have not figured out yet how to create tests that check that the
right SQL is being generated for the backend used. (Nor how to test
the right database was touched by an action, this is quite obvious
manually, but I do not know how to automate this.)

I put some ideas on using multiple databases for copying (backup or
migration) of objects (point 4 of the discussion note) in [http://
trac.woe-beti.de/wiki/APIModelDuplication].

Please comment, add, shoot etc. Any help will be much appreciated.

Koen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ivan Sagalaev]

Simon Willison wrote:
> How about mysql_masterslave or mysql_replicated (I prefer the second)?

Yes, mysql_replicated seems right. Thanks!

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Manuel Saelices]

On 23 mayo, 13:00, "Mike Scott" <[EMAIL PROTECTED]> wrote:
>
> Maybe having to state a storage location on a per-row level. (IE this could
> happen by overriding the manager, and simply switching DB at selection time.
> or being able to provide the DB info at selection time.)

Maybe i good thing was to provide a middleware that does db selection.
Think in applications user centered. In those, user define what DB
use.

You can provide an API for change DB of all next queries, and use if
you want in a middleware (looking at request.user).
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Simon Willison]

On May 22, 6:53 pm, Ivan Sagalaev <[EMAIL PROTECTED]> wrote:
> Simon Willison wrote:
> > Thankfully Ivan Sagalaev's confusingly named mysql_cluster
>
> BTW does anyone have a suggestion how to rename it? I've picked
> mysql_cluster simply because I didn't know that there exists the thing
> named "MySQL Cluster" (no kidding :-) ).

How about mysql_masterslave or mysql_replicated (I prefer the second)?
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[koenb]

You need to be aware that this is quite complicated if your model has
foreign keys in it: if you use the ORM to do queries, the ORM would
have to be so smart as to when to split up your queries instead of
doing joins.
eg you have model A which foreign keys to a User model. For a row of A
that is in the same database as User, the ORM could simply use a join,
but for a row of A that was already in the other database, it can't.

I do not believe this is a trivial change.

My proposal is to keep things simple in a first phase: let's make it
possible to use different databases for different models with the
restriction that relations should not cross databases. Once we get all
that working, we may look at making the query generation deal with
those.

Koen

On 23 mei, 13:00, "Mike Scott" <[EMAIL PROTECTED]> wrote:
> On Fri, May 23, 2008 at 2:59 AM, Simon Willison <[EMAIL PROTECTED]>
> wrote:
>
> > 1. Replication - being able to send all of my writes to one master
> > machine but spread all of my reads over several slave machines.
> > Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
> > problem neatly without modification to Django core - it's just an
> > alternative DB backend which demonstrates that doing this isn't
> > particularly hard:http://softwaremaniacs.org/soft/mysql_cluster/en/
>
> Personally I think this is something that is better solved by the database
> software itself. Having replication code-side is something that I don't feel
> to good about. But maybe thats just me.
>
>
>
> > 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> > User entries 1001-2000 live on DB2 and so on.
>
> This is something I would love, an example being archives. (As Eratothene
> points out.
>
> Maybe having to state a storage location on a per-row level. (IE this could
> happen by overriding the manager, and simply switching DB at selection time.
> or being able to provide the DB info at selection time.)
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Mike Scott]

On Fri, May 23, 2008 at 2:59 AM, Simon Willison <[EMAIL PROTECTED]>
wrote:

> 1. Replication - being able to send all of my writes to one master
> machine but spread all of my reads over several slave machines.
> Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
> problem neatly without modification to Django core - it's just an
> alternative DB backend which demonstrates that doing this isn't
> particularly hard: http://softwaremaniacs.org/soft/mysql_cluster/en/


Personally I think this is something that is better solved by the database
software itself. Having replication code-side is something that I don't feel
to good about. But maybe thats just me.


>
>
> 2. Sharding - being able to put User entries 1-1000 on DB1, whereas
> User entries 1001-2000 live on DB2 and so on.
>
>
>
This is something I would love, an example being archives. (As Eratothene
points out.

Maybe having to state a storage location on a per-row level. (IE this could
happen by overriding the manager, and simply switching DB at selection time.
or being able to provide the DB info at selection time.)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Eratothene]

I think there is a third issue.

Usage of several RDBMS in one django application simulatneously

For example we maintain two RDBMS: monetdb and postgresql. The latest
and most accessed data is stored in monetdb for performance. After one
month data is moved to posgresql which holds the full archive.


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ivan Sagalaev]

Simon Willison wrote:
> Thankfully Ivan Sagalaev's confusingly named mysql_cluster

BTW does anyone have a suggestion how to rename it? I've picked 
mysql_cluster simply because I didn't know that there exists the thing 
named "MySQL Cluster" (no kidding :-) ).

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ben Ford]

Hi all,

I've now re-applied Daryls patch (which was against qsrf) to a clone of
django trunk in a mercurial repo. It's available at
http://hg.woe-beti.deand there's a trac set up for it at
http://trac.woe-beti.de. Feel free to make use of both of these. Although
I've disabled to ability to create tickets perhaps the wiki might be a good
place to discuss the API? Anyone can clone from the hg repo, give me a shout
if you would like push access and I'll sort it out.

Cheers,
Ben

-- 
Regards,
Ben Ford
[EMAIL PROTECTED]
+447792598685

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Simon Willison]

I have to admit I'm slightly worried about the multi-database
proposal, because at the moment it doesn't seem to solve either of the
multi-db problems I'm concerned about.

The proposal at the moment deals with having different models live in
different databases - for example, the Forum application lives on DB1
while the Blog application lives on DB2.

I can see how this could be useful, but the two database problems that
keep me up at night are the following:

1. Replication - being able to send all of my writes to one master
machine but spread all of my reads over several slave machines.
Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
problem neatly without modification to Django core - it's just an
alternative DB backend which demonstrates that doing this isn't
particularly hard: http://softwaremaniacs.org/soft/mysql_cluster/en/

2. Sharding - being able to put User entries 1-1000 on DB1, whereas
User entries 1001-2000 live on DB2 and so on.

I'd love Django to have built-in abilities to solve #1 - it's a really
important first-step on scaling up to multiple databases, and it's
also massively easier than any other part of the multi-db problem.

I wouldn't expect a magic solution to #2 because it's so highly
dependent on the application that is being built, but at the same time
it would be nice to see a multi-db solution at least take this in to
account (maybe just by providing an easy tool to direct an ORM request
to a specific server based on some arbitrary logic).

I may have misunderstood the proposal, but I think it's vital that the
above two use cases are considered. Even if they can't be solved
outright, providing tools that custom solutions to these cases can be
built with should be a priority for multi-db support.

Cheers,

Simon
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[koenb]

I really like this line of thought: having the persistence layer of a
model fixed is a good idea.
Relations is a big issue here: unless we can support relations across
databases, switching connections is always a big opportunity to shoot
yourself in the foot.
I would propose a function that can collect "clusters" of models, that
is a collection of models that somehow are related to each other and
use that function to a) check that they all use the same database
during validation, and b) if we provide a API to register a model for
an additional connection (that is a second one), you get copies of the
models for the entire cluster, relations and all. Like that we could
even have syncdb create the tables for these 'backup models' too.

Koen

On 21 mei, 23:16, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]>
wrote:
> Hi guys --
>
> Sorry for coming late to the party, but I'm just now catching up on 
> django-dev.
>
> I'm really glad to see you get the ball rolling on multiple db
> support, and once I'm dug out from my backlog I'll be happy to start
> reviewing code and helping out if I'm needed.
>
> However, before we get to that point, I've got some pretty serious API
> concerns with the current approach, so I think I should outline those
> before y'all go much further. I don't want you to expend much effort
> just to get a -1 smackdown.
>
> The current mechanism of defining "other" databases in the settings
> module is just fine, and the underlying mechanism of having
> queries/managers "know" their connection is similarly dandy. But the
> wheels come off when it comes to the "public" API where users will
> choose which connection they use.
>
> As far as I can tell, you've currently provided two hooks to use a
> secondary connection: set the model's default connection in the
> settings module (which is OK, I suppose, though I might want to
> nitpick the syntax a bit), and assigning to ``Model.objects.db``.
>
> This second one is a disaster waiting to happen -- you've had to muddy
> things up with threadlocals to work around some problems already. Also
> consider the "bookkeeping" you'd need to do to deal with objects
> across multiple database simultaneously (think sharding). You'd have
> to keep juggling ``Model.objects.db`` and saving old ones... ugh.
>
> Here's how I think it should work:
>
> * I'd like the default connection for each and every object to be the
> default database forever and always. I find putting models for default
> connections in settings distasteful and I'd rather just a single API
> for changing the connection (see below). However, I imagine I'll be in
> the minority here so I'm prepared to cede this point if necessary.
>
> * There needs to be an official API to get a model (or perhaps a
> manager) which references a different "context" --
> ``Model.objects.db`` should be read-only. So you'd call some API
> method, and get back a sort of proxy object that uses the other
> connection. Here's a strawman API::
>
> >>> from django import db
> >>> from someapp.models import Article
>
> >>> Article.objects.all()
> [... all Articles from the default database ...]
>
> >>> ArticlesOnOtherDatabase =
> db.get_model_for_other_connection(Article, "private")
> >>> ArticlesOnOtherDatabase.objects.all()
> [... all Articles from the database defined with the "private" key ...]
>
> This should make the threadlocal stuff unnecessary, and (to my eye) is
> a lot more sane than assigning the ``Manager.db``. Oh, and please
> choose a better better name than
> ``db.get_model_for_other_connection()``; given that you're building
> the bikeshed you might as well paint it, too.
>
> Jacob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[oggie rob]

On May 21, 2:16 pm, "Jacob Kaplan-Moss" <[EMAIL PROTECTED]>
wrote:
> * There needs to be an official API to get a model (or perhaps a
> manager) which references a different "context" --
> ``Model.objects.db`` should be read-only. So you'd call some API
> method, and get back a sort of proxy object that uses the other
> connection. Here's a strawman API::
>
>     >>> from django import db
>     >>> from someapp.models import Article
>
>     >>> Article.objects.all()
>     [... all Articles from the default database ...]
>
>     >>> ArticlesOnOtherDatabase =
> db.get_model_for_other_connection(Article, "private")
>     >>> ArticlesOnOtherDatabase.objects.all()
>     [... all Articles from the database defined with the "private" key ...]
>

Has anybody considered declaring the connection when getting the
manager? Something like:
Artist.objects.all()
Widget.objects(db='a').all()
Obviously with the default database for the case when "db" isn't
passed. Also you could override the Manager to use a different
database by default (e.g. Widget.objects.all() might always use an
OTHER_DATABASE while all other models use the main db, if you create a
custom Manager for Widget)

This still leaves questions about how syncdb would be achieved, at
least. But if it could be done, the API seems simple to understand.

 -rob
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ben Ford]

Hi Jacob,

I'd be interested in your thoughts on a declarative approach to defining the
other databases...? I'll have my mercurial repo synced up to trunk tomorrow
(my time) and I'll re-apply the patch I got from Daryl to it as a starting
point. Hopefully people will be able to have a look through it and compare
the declarative approach proposed with the existing multi-db approach.

 As far as I can tell, you've currently provided two hooks to use a
> secondary connection: set the model's default connection in the
> settings module (which is OK, I suppose, though I might want to
> nitpick the syntax a bit), and assigning to ``Model.objects.db``.
>
> This second one is a disaster waiting to happen -- you've had to muddy
> things up with threadlocals to work around some problems already. Also
> consider the "bookkeeping" you'd need to do to deal with objects
> across multiple database simultaneously (think sharding). You'd have
> to keep juggling ``Model.objects.db`` and saving old ones... ugh.


I built a non trivial application with multi-db as it is right now and found
the api to be a bit hairy to be honest. I think it would be an advantage,
especially in a "database rich" environment to be able to build a db on the
fly much like a model, rather than be tied to what's in a dict in settings.
I don't really like the objects.db way of doing it, and I found myself doing
a fair bit of hacking to get it to work.

> * There needs to be an official API to get a model (or perhaps a
> manager) which references a different "context" --
> ``Model.objects.db`` should be read-only. So you'd call some API
> method, and get back a sort of proxy object that uses the other
> connection. Here's a strawman API::
>
>  >>> from django import db
>  >>> from someapp.models import Article
>
>  >>> Article.objects.all()
>   [... all Articles from the default database ...]
>
>  >>> ArticlesOnOtherDatabase = db.get_model_for_other_connection(Article,
"private")
>  >>> ArticlesOnOtherDatabase.objects.all()
>  [... all Articles from the database defined with the "private" key ...]

Agreed, the way I got round this was to build the model again from scratch
each time I wanted to access objects in a different database and have the
dynamicaly created model persist in the app cache. I took most of this from
the dynamic models entry on the wiki, it's here, look in the duplicate_model
function:
http://www.djangosnippets.org/snippets/442/
This would really need work (especially the field copying code, which is
fairly horrifying! I know that doesn't work for all field types too - yuk)
before it becomes a 'good idea', and I'm not even sure it's the right way to
go, however I'd be interested in weather people think it's a valid approach.

 * I'd like the default connection for each and every object to be the
> default database forever and always. I find putting models for default
> connections in settings distasteful and I'd rather just a single API
> for changing the connection (see below). However, I imagine I'll be in
> the minority here so I'm prepared to cede this point if necessary.


The API which I think is being proposed is that there should be a central
register for database connections. In my mind this would be the place to go
to get hold of a connection for use in a queryset (and all the other places
it's needed) and I think the correct default behaviour of the class/object
would be to return the connection defined in settings.DATABASE_*. The code
to build the declarative DatabaseWrapper is already there, and there a
method to build one of these from what's in settings too. This should make
it easy to get hold of connection in all of the places where we currently do
"from django.db import connection".

It's great to see this revived again :-)

Cheers
Ben


-- 
Regards,
Ben Ford
[EMAIL PROTECTED]
+447792598685

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Jacob Kaplan-Moss]

Hi guys --

Sorry for coming late to the party, but I'm just now catching up on django-dev.

I'm really glad to see you get the ball rolling on multiple db
support, and once I'm dug out from my backlog I'll be happy to start
reviewing code and helping out if I'm needed.

However, before we get to that point, I've got some pretty serious API
concerns with the current approach, so I think I should outline those
before y'all go much further. I don't want you to expend much effort
just to get a -1 smackdown.

The current mechanism of defining "other" databases in the settings
module is just fine, and the underlying mechanism of having
queries/managers "know" their connection is similarly dandy. But the
wheels come off when it comes to the "public" API where users will
choose which connection they use.

As far as I can tell, you've currently provided two hooks to use a
secondary connection: set the model's default connection in the
settings module (which is OK, I suppose, though I might want to
nitpick the syntax a bit), and assigning to ``Model.objects.db``.

This second one is a disaster waiting to happen -- you've had to muddy
things up with threadlocals to work around some problems already. Also
consider the "bookkeeping" you'd need to do to deal with objects
across multiple database simultaneously (think sharding). You'd have
to keep juggling ``Model.objects.db`` and saving old ones... ugh.

Here's how I think it should work:

* I'd like the default connection for each and every object to be the
default database forever and always. I find putting models for default
connections in settings distasteful and I'd rather just a single API
for changing the connection (see below). However, I imagine I'll be in
the minority here so I'm prepared to cede this point if necessary.

* There needs to be an official API to get a model (or perhaps a
manager) which references a different "context" --
``Model.objects.db`` should be read-only. So you'd call some API
method, and get back a sort of proxy object that uses the other
connection. Here's a strawman API::

>>> from django import db
>>> from someapp.models import Article

>>> Article.objects.all()
[... all Articles from the default database ...]

>>> ArticlesOnOtherDatabase =
db.get_model_for_other_connection(Article, "private")
>>> ArticlesOnOtherDatabase.objects.all()
[... all Articles from the database defined with the "private" key ...]

This should make the threadlocal stuff unnecessary, and (to my eye) is
a lot more sane than assigning the ``Manager.db``. Oh, and please
choose a better better name than
``db.get_model_for_other_connection()``; given that you're building
the bikeshed you might as well paint it, too.

Jacob

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Daryl Spitzer]

> Is there a ticket in django we could use to track progress on this? We could
> use 4747, but if we do decide on a new API that might be a bit confusing...
> We could of course just use the mailing list and trac project, thoughts?

There's also http://code.djangoproject.com/ticket/1142.  With the
mailing list and trac project, do we need a ticket for more than just
a place to attach patches to invite others to test?

> I'll sort out the hg repo (it now needs to point at trunk - not qsrf) and
> trac project if I get time this evening and make it public readable for
> everyone who's interested.

Thanks Ben.

--
Daryl


On Wed, May 21, 2008 at 3:33 AM, Ben Ford <[EMAIL PROTECTED]> wrote:
> Hi all,
>
> I'll sort out the hg repo (it now needs to point at trunk - not qsrf) and
> trac project if I get time this evening and make it public readable for
> everyone who's interested.
>
> Is there a ticket in django we could use to track progress on this? We could
> use 4747, but if we do decide on a new API that might be a bit confusing...
> We could of course just use the mailing list and trac project, thoughts?
>
> It's great to see some interest in multiple db support again :-)
>
> Ben
>
> 2008/5/20 Nicola Larosa (tekNico) <[EMAIL PROTECTED]>:
>>
>> Daryl Spitzer wrote:
>> > If I don't, I see if I can at least make enough time to write up the API
>> > I came up with at PyCon.
>>
>> Please do, that would be great.
>>
>> --
>> Nicola Larosa - http://www.teknico.net/
>>
>>
>
>
>
> --
> Regards,
> Ben Ford
> [EMAIL PROTECTED]
> +447792598685
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Ben Ford]

Hi all,

I'll sort out the hg repo (it now needs to point at trunk - not qsrf) and
trac project if I get time this evening and make it public readable for
everyone who's interested.

Is there a ticket in django we could use to track progress on this? We could
use 4747, but if we do decide on a new API that might be a bit confusing...
We could of course just use the mailing list and trac project, thoughts?

It's great to see some interest in multiple db support again :-)

Ben

2008/5/20 Nicola Larosa (tekNico) <[EMAIL PROTECTED]>:

>
> Daryl Spitzer wrote:
> > If I don't, I see if I can at least make enough time to write up the API
> > I came up with at PyCon.
>
> Please do, that would be great.
>
> --
> Nicola Larosa - http://www.teknico.net/
>
> >
>


-- 
Regards,
Ben Ford
[EMAIL PROTECTED]
+447792598685

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Nicola Larosa (tekNico)]

Daryl Spitzer wrote:
> If I don't, I see if I can at least make enough time to write up the API
> I came up with at PyCon.

Please do, that would be great.

--
Nicola Larosa - http://www.teknico.net/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Daryl Spitzer]

I've unfortunately been too busy to make time to work on this since
PyCon.  The last thing I've done (after writing some code on the
flight home) is to send a patch to Ben Ford.  Not long after that Ben
created a Mercurial repository (with my patch) and a Trac project.
You'll want to contact him.

I would still like to get my patch working so others (and myself) can
start testing it.  I won't have time this week, but so far it looks
like I may be able to make some time next week.  If I don't, I see if
I can at least make enough time to write up the API I came up with at
PyCon.

--
Daryl


On Tue, May 20, 2008 at 8:05 AM, Casper Jensen <[EMAIL PROTECTED]> wrote:
>
> On Tue, May 20, 2008 at 4:56 PM, Nicola Larosa (tekNico)
> <[EMAIL PROTECTED]> wrote:
>> It would be nice to coordinate each one's efforts, to avoid wasting
>> time. Ben, Daryl, any news?
> Currently, I have not worked on the project, since the proposal,
> because of job and university commitments. I plan to track the
> development at begin to help with the development when I get more time
> (over the summer).
>
> - Casper
>
> >
>

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Casper Jensen]

On Tue, May 20, 2008 at 4:56 PM, Nicola Larosa (tekNico)
<[EMAIL PROTECTED]> wrote:
> It would be nice to coordinate each one's efforts, to avoid wasting
> time. Ben, Daryl, any news?
Currently, I have not worked on the project, since the proposal,
because of job and university commitments. I plan to track the
development at begin to help with the development when I get more time
(over the summer).

- Casper

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[koenb]

Ah, missed that one.
Anyway, I only did the easy parts (that is, getting data in and out of
existing databases).
Thanks for the pointer, I'll try to keep an eye on that.

Koen

On 20 mei, 16:56, "Nicola Larosa (tekNico)" <[EMAIL PROTECTED]>
wrote:
> koenb wrote:
> > For those interested in multiple database support, I have started
> > working on it again, and posted my work-in-progress to ticket #4747.
> > ...
> > Anyway, if anyone is interested in helping, please let me know!
>
> I am going to need this in a month or so. Actions speak louder than
> words, so many thanks for your efforts. However, there were news two
> months ago, summarized in this thread:
>
> Yet another SoC introduction: Getting multi-db 
> donehttp://groups.google.com/group/django-developers/browse_thread/thread...
>
> It would be nice to coordinate each one's efforts, to avoid wasting
> time. Ben, Daryl, any news?
>
> --
> Nicola Larosa -http://www.teknico.net/
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support

[Nicola Larosa (tekNico)]

koenb wrote:
> For those interested in multiple database support, I have started
> working on it again, and posted my work-in-progress to ticket #4747.
> ...
> Anyway, if anyone is interested in helping, please let me know!

I am going to need this in a month or so. Actions speak louder than
words, so many thanks for your efforts. However, there were news two
months ago, summarized in this thread:

Yet another SoC introduction: Getting multi-db done
http://groups.google.com/group/django-developers/browse_thread/thread/a0bc69e64ad8e318/

It would be nice to coordinate each one's efforts, to avoid wasting
time. Ben, Daryl, any news?

--
Nicola Larosa - http://www.teknico.net/

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Multiple database support

[koenb]

For those interested in multiple database support, I have started
working on it again, and posted my work-in-progress to ticket #4747.

I started from trunk and added things from the multidb branch little
by little, since so much has changed in that area since then.
There is still a lot more that needs to be checked and a number of
things to be redone. References to the default connection object are
all over the codebase, there is still a lot of work left to get all of
them straightened out.

What is working (more or less):
- using existing databases (though there might still be quirks when
using different engines)
- running tests
- some of the management commands (eg sqlall, sqlflush, but loaddata,
inspectdb or syncdb are not quite there yet)

Important to mention is that relations across databases are not
supported.

Anyway, if anyone is interested in helping, please let me know!

Koen
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: multiple database support and different DB types

[[EMAIL PROTECTED]]

yep, tried that.

On Jun 7, 6:54 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
wrote:
> I haven't tried this at all myself, as I just use mysql ...but are you
> saying that specifying a different DATABASE_ENGINE in the
> OTHER_DATABASES entries doesn't work?  Was just making sure you had
> tried that.
>
> On Jun 7, 2:02 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
>
> > hi, i'm using the multiple DB support branch and i'd like to have one
> > DB be MySQL and one be sqlite, but i can't, because all of django's
> > generated SQL conforms to the primary DB type. so in this case all the
> > queries sent to the sqlite DB have backticks, which causes parsing
> > errors.
>
> > can anyone suggest where i should start if i want to patch in the
> > ability to generate different SQL for each DB depending on its type?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: multiple database support and different DB types

[[EMAIL PROTECTED]]

I haven't tried this at all myself, as I just use mysql ...but are you
saying that specifying a different DATABASE_ENGINE in the
OTHER_DATABASES entries doesn't work?  Was just making sure you had
tried that.

On Jun 7, 2:02 pm, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote:
> hi, i'm using the multiple DB support branch and i'd like to have one
> DB be MySQL and one be sqlite, but i can't, because all of django's
> generated SQL conforms to the primary DB type. so in this case all the
> queries sent to the sqlite DB have backticks, which causes parsing
> errors.
>
> can anyone suggest where i should start if i want to patch in the
> ability to generate different SQL for each DB depending on its type?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

multiple database support and different DB types

[[EMAIL PROTECTED]]

hi, i'm using the multiple DB support branch and i'd like to have one
DB be MySQL and one be sqlite, but i can't, because all of django's
generated SQL conforms to the primary DB type. so in this case all the
queries sent to the sqlite DB have backticks, which causes parsing
errors.

can anyone suggest where i should start if i want to patch in the
ability to generate different SQL for each DB depending on its type?


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142): branch?

[[EMAIL PROTECTED]]

Jacob,

I sent you an email about getting commit access to the branch a couple
of days ago. Did you get it, or should I resend?

Thanks,

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142): branch?

[Jacob Kaplan-Moss]

On Jun 23, 2006, at 9:16 AM, [EMAIL PROTECTED] wrote:
> So, would it be possible to create a branch for work on multiple db
> support? If so, and if you can restrict commits to branches only, then
> it would be great for Jon and myself to have commit access there.

Created at http://code.djangoproject.com/svn/django/branches/multiple- 
db-support/.

Please contact me off-list for commit access to that branch; find me  
at this email address or on irc.freenode.net as "jacobkm".

Jacob

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Multiple database support (#1142): branch?

[[EMAIL PROTECTED]]

I've been hacking away at #1142 (multiple database support) a bit more,
with  some help from Jon Dugan. As you can see from the patches
attached to the wiki page:
http://code.djangoproject.com/wiki/MultipleDatabaseSupport, the changes
needed to support multiple dbs are many and far-reaching. While there
*should* be no impact on backwards compatibility, I would be pretty
uncomfortable asking for a patch of this size to be committed to trunk.

So, would it be possible to create a branch for work on multiple db
support? If so, and if you can restrict commits to branches only, then
it would be great for Jon and myself to have commit access there.

Thanks!

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[David Elias]

[EMAIL PROTECTED] wrote:
> perhaps we could put it all into backend/creation.py.

That's what i had, early, implemented with the firebird backend patch,
put the sequence and triggers sql in creation.py. And now i was trying
retain the management.py intact in terms of functionality, but your
aproach in puting all the sql statements in creation module is better,
i think. At least will give much more flexibility for generating SQL
for all backends.

Nice work.

David


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[[EMAIL PROTECTED]]

Wiki page up:

http://code.djangoproject.com/wiki/MultipleDatabaseSupport

The current (sloppy, totally incomplete) patch is attached, along with
a description of where I see it heading.

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[Adrian Holovaty]

On 6/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I'll start working in that direction tomorrow, if that seems like a
> good plan. I'm going to be mostly internetless for the next 2 weeks or
> so, so it will be a while before I can actually submit a patch that's
> fully functional. But I can mail or post up what I have so far tomorrow
> morning, if there's any interest. Would it be worthwhile to start a
> wiki page at this point?

Sounds good -- looking forward to it! Yes, go ahead and start a wiki
page if you get a chance.

Adrian

-- 
Adrian Holovaty
holovaty.com | djangoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[[EMAIL PROTECTED]]

> As I mentioned in another note to this thread, things are the way they
> are because I didn't want to load all the rarely-used reflection stuff
> into memory each time a model is used. That said, if it helps your
> goal (which would be a great Django addition), let's go ahead and make
> those model methods. One immediate problem I'd see is that it
> increases the number of reserved words (assuming these are model-level
> class methods). Perhaps, like "objects" for Managers, we should put
> all the reflection stuff within the namespace of a single attribute.
> Or maybe they become Manager methods, to avoid that problem entirely?

Excellent points. I like the idea of putting the schema manipulation
functions in the manager -- much better than polluting the model
namespace. (Heck, I think save() and delete() should go in there too,
but that's another issue entirely... ). To avoid loading all of this
logic when it's not needed, perhaps we could put it all into
backend/creation.py. That might imply creating something like an
ansisql backend with just common creation parts, since so much is
similar across different databases. (SQLAlchemy takes a similar
approach -- a great project with many good, stealable ideas. :)

I'll start working in that direction tomorrow, if that seems like a
good plan. I'm going to be mostly internetless for the next 2 weeks or
so, so it will be a while before I can actually submit a patch that's
fully functional. But I can mail or post up what I have so far tomorrow
morning, if there's any interest. Would it be worthwhile to start a
wiki page at this point?

Thanks for the great feedback, all.

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[[EMAIL PROTECTED]]

> And, in any case, they need to know the table name (and quite
> possibly database name and connection proxy) for the related tables. So
> you are going to have do a pass through all the models and build up the
> graph of dependencies and make that available to each model at
> construction time as well, aren't you?

Sure, but that has to happen anyway to support initial data for models
that have foreign keys. It won't be too difficult to produce a list of
models for an app ordered by dependency. Actually, hasn't someone
already done the inverse (producing models in dependency order) with
inspectdb? The tough part will be inter-app dependencies. Probably best
to handle those as django does now, with a pending_references set, but
I need to mull that over for a while.

> [As an aside: I think we are also going to discover that ForeignKey is
> unfortunately named, because one-to-many relations to tables in another
> database is not complete nonsense; I've worked on systems in the past
> that separated frequently read tables from frequently updated, less
> frequently read tables for performance reasons.]

The crunchy PHP ORM that I wrote at my day job and that I fervently
hope to retire, along with the rest of the equally-crunch framework
surrounding it, supports "AlienForeignKey" and "AlienManyToMany"
relations -- it's quite easy when you can depend on the alien model to
do most of the work.

> Nice work. :-)

Thanks! Hopefully it will remain nice when it's done. :)

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[Adrian Holovaty]

On 6/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I'd argue that the right solution here would be to push the brains
> farther out to the edge. Have management functions call class methods
> on models to execute table creation, initial data loading, etc, rather
> than having them poll the models for information and construct and
> execute the sql themselves.

As I mentioned in another note to this thread, things are the way they
are because I didn't want to load all the rarely-used reflection stuff
into memory each time a model is used. That said, if it helps your
goal (which would be a great Django addition), let's go ahead and make
those model methods. One immediate problem I'd see is that it
increases the number of reserved words (assuming these are model-level
class methods). Perhaps, like "objects" for Managers, we should put
all the reflection stuff within the namespace of a single attribute.
Or maybe they become Manager methods, to avoid that problem entirely?

Adrian

-- 
Adrian Holovaty
holovaty.com | djangoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[Malcolm Tredinnick]

On Wed, 2006-06-07 at 22:25 +, [EMAIL PROTECTED] wrote:

[...]
> However, I've run into a problem that can't be fixed with little
> patches, in django.core.management. Specifically, all of the many
> get_sql_* functions in there pull together sql from multiple models and
> execute it all with the default connection. That's not going to work if
> model A and model B want different connections.
> 
> I'd argue that the right solution here would be to push the brains
> farther out to the edge. Have management functions call class methods
> on models to execute table creation, initial data loading, etc, rather
> than having them poll the models for information and construct and
> execute the sql themselves. Something like:
> 
> def install(app):
> # ... validation, start a transaction, etc
> for model in models.get_models(app):
> model.install()
> 
> Rather than the current:
> 
> def install(app):
> # ...
> sql_list = get_sql_all(app)
> 
> try:
> cursor = connection.cursor()
> for sql in sql_list:
> cursor.execute(sql)
> 
> That would be a pretty substantial change, but I think it would open up
> a lot of interesting possiblities for models that behave differently
> from the default -- and it certainly would make supporting multiple
> databases a whole lot easier.

I don't think you can completely remove the controller portion here,
although you can push a lot of the mechanics down into the model
managers. The difficulty is that models do not exist completely
independently of other models.

Think about relations between models. ForeignKey and friends now have to
be implemented differently or can only apply to models using the same
database. And, in any case, they need to know the table name (and quite
possibly database name and connection proxy) for the related tables. So
you are going to have do a pass through all the models and build up the
graph of dependencies and make that available to each model at
construction time as well, aren't you?

[As an aside: I think we are also going to discover that ForeignKey is
unfortunately named, because one-to-many relations to tables in another
database is not complete nonsense; I've worked on systems in the past
that separated frequently read tables from frequently updated, less
frequently read tables for performance reasons.]

> 
> What do you all think?

Nice work. :-)

Best wishes,
Malcolm


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[Adrian Holovaty]

On 6/7/06, Jeremy Dunck <[EMAIL PROTECTED]> wrote:
> On 6/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> > I'd argue that the right solution here would be to push the brains
> > farther out to the edge. Have management functions call class methods
> > on models to execute table creation, initial data loading, etc, rather
> > than having them poll the models for information and construct and
> > execute the sql themselves. Something like:
>
> I don't deserve much of a vote, but when I peaked in manage.py, I
> thought the same thing.  Wha?  All that reflection magic isn't in the
> model itself?

The reason for that is that the reflection magic is only needed in
special situations -- namely, table creation -- so there's no need to
load it into memory.

Adrian

-- 
Adrian Holovaty
holovaty.com | djangoproject.com

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Re: Multiple database support (#1142) roadblock

[Jeremy Dunck]

On 6/7/06, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
> I'd argue that the right solution here would be to push the brains
> farther out to the edge. Have management functions call class methods
> on models to execute table creation, initial data loading, etc, rather
> than having them poll the models for information and construct and
> execute the sql themselves. Something like:

I don't deserve much of a vote, but when I peaked in manage.py, I
thought the same thing.  Wha?  All that reflection magic isn't in the
model itself?

But, yeah, it was too big a thing for me to start.  ;-)

--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---

Multiple database support (#1142) roadblock

[[EMAIL PROTECTED]]

I've been hacking away today at a patch for #1142 (multiple database
connection support), which is the last serious technical hurdle to my
company's use of django -- a very itchy itch for me to scratch.

Mostly things have gone well. My basic design is to add, as a
supplement to the current db.connection, db.connections: a lazy
connection manager that acts like a dict of named connections. It picks
up its settings for those connections from a new settings var,
DATABASES, like so:

>>> from django.conf import settings
>>> settings.DATABASES = {
... 'a': { 'DATABASE_ENGINE': 'sqlite3',
...'DATABASE_NAME': '/tmp/dba.db'
... },
... 'b': { 'DATABASE_ENGINE': 'sqlite3',
...'DATABASE_NAME': '/tmp/dbb.db'
... }}
>>> from django.db import connections

# connections[database] holds an object that includes connection,
# DatabaseError, backend, get_introspection_module,
# get_creation_module, and runshell. Connections are established only
# when requested.

>>> connections['a']
Connection:  (ENGINE=sqlite3 NAME=/tmp/dba.db)
>>> connections['b']
Connection:  (ENGINE=sqlite3 NAME=/tmp/dbb.db)

(That's right out of the doctests for the new code.)

This, plus a little patching of Meta options gets models their own
connections, set by a Meta option, and used by managers attached to the
model. I still have some work to do to integrate with the transactions
module.

However, I've run into a problem that can't be fixed with little
patches, in django.core.management. Specifically, all of the many
get_sql_* functions in there pull together sql from multiple models and
execute it all with the default connection. That's not going to work if
model A and model B want different connections.

I'd argue that the right solution here would be to push the brains
farther out to the edge. Have management functions call class methods
on models to execute table creation, initial data loading, etc, rather
than having them poll the models for information and construct and
execute the sql themselves. Something like:

def install(app):
# ... validation, start a transaction, etc
for model in models.get_models(app):
model.install()

Rather than the current:

def install(app):
# ...
sql_list = get_sql_all(app)

try:
cursor = connection.cursor()
for sql in sql_list:
cursor.execute(sql)

That would be a pretty substantial change, but I think it would open up
a lot of interesting possiblities for models that behave differently
from the default -- and it certainly would make supporting multiple
databases a whole lot easier.

What do you all think?

Thanks,

JP


--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups 
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at 
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---