[#20291] Add method to reload `AppCache`
Hi, Speaking of: https://code.djangoproject.com/ticket/20291 I don't see how is this ticket related to #3591 (which BTW says "add support for custom app_label and verbose_name"). This one is a little atomic piece of work and a handy addition that I need as well, and waiting for a ticket that is already six years old to be closed some day means that it may happen when I'm retired. So, what do you think about reopening this one? Greetings, Robert -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Changing deferred model attribute behavior
At the moment, if you call defer() or only() on a QuerySet, then access the
deferred fields individually, *each* access of a deferred field will result
in a separate query.
For example, assuming a User model with username/bio/location fields, this
is what currently happens:
"""
>>> u = User.objects.only('username').get(id=1)
# Results in: SELECT id, username FROM users WHERE id=1
>>> u.bio
# Results in SELECT bio FROM users WHERE id=1
>>> u.location
# Results in SELECT location FROM users WHERE id=1
"""
I'd like there to be a way of retrieving *all* deferred fields the first
time a deferred field is accessed. So with the above example, this would
happen instead:
"""
>>> u = User.objects.only('username').get(id=1)
# Results in: SELECT id, username FROM users WHERE id=1
>>> u.bio
# Results in SELECT bio, location FROM users WHERE id=1
>>> u.location
# No database query
"""
Here's my use case. In my app, I'm storing frequently accessed user
attributes (id, username, etc.) in signed cookies, to prevent an
unnecessary database hit on every page view. I create a User object with
those attributes, so that I can take advantage of various methods on the
User class, but I want the other User fields to be lazily loaded. I have
this all working except for the last piece, which is to change lazy loading
such that it loads *everything* else the first time a deferred field is
accessed.
So that's the "what" and "why" -- here's the how...
The current implementation is in django/db/models/query_utils.py -- see
the DeferredAttribute class. When you call only() or defer() on a QuerySet,
the resulting model instance will have a DeferredAttribute instance for
each deferred field. The problem is that DeferredAttributes only load their
own column's data, not the data for all *other* DeferredAttributes on the
given model instance. Hence, the individual SQL queries for each column.
To solve this, we would need to change DeferredAttribute to find all
*other* DeferredAttributes on the given model and load them in a single
query somehow.
Also, I should mention that this should be *optional* behavior, as the
current behavior is reasonable for the common case. The API for specifying
this "load everything" behavior is a separate discussion. Perhaps a keyword
argument like: User.objects.only('username', loadall=True).
Thoughts?
Adrian
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
This sounds like a reasonable request, I don't yet have an opinion on API
or anything. One tiny thing I'd like to note though, "change DeferredAttribute
to find all *other* DeferredAttributes". I don't think `finding` is the
right way to think about it, a `DeferredAttribute` with loadall semantics
should know about the specific set of objects that came from the queryset.
Alex
On Thu, Apr 25, 2013 at 12:06 PM, Adrian Holovaty wrote:
> At the moment, if you call defer() or only() on a QuerySet, then access
> the deferred fields individually, *each* access of a deferred field will
> result in a separate query.
>
> For example, assuming a User model with username/bio/location fields, this
> is what currently happens:
>
> """
> >>> u = User.objects.only('username').get(id=1)
> # Results in: SELECT id, username FROM users WHERE id=1
>
> >>> u.bio
> # Results in SELECT bio FROM users WHERE id=1
>
> >>> u.location
> # Results in SELECT location FROM users WHERE id=1
> """
>
> I'd like there to be a way of retrieving *all* deferred fields the first
> time a deferred field is accessed. So with the above example, this would
> happen instead:
>
> """
> >>> u = User.objects.only('username').get(id=1)
> # Results in: SELECT id, username FROM users WHERE id=1
>
> >>> u.bio
> # Results in SELECT bio, location FROM users WHERE id=1
>
> >>> u.location
> # No database query
> """
>
> Here's my use case. In my app, I'm storing frequently accessed user
> attributes (id, username, etc.) in signed cookies, to prevent an
> unnecessary database hit on every page view. I create a User object with
> those attributes, so that I can take advantage of various methods on the
> User class, but I want the other User fields to be lazily loaded. I have
> this all working except for the last piece, which is to change lazy loading
> such that it loads *everything* else the first time a deferred field is
> accessed.
>
> So that's the "what" and "why" -- here's the how...
>
> The current implementation is in django/db/models/query_utils.py -- see
> the DeferredAttribute class. When you call only() or defer() on a QuerySet,
> the resulting model instance will have a DeferredAttribute instance for
> each deferred field. The problem is that DeferredAttributes only load their
> own column's data, not the data for all *other* DeferredAttributes on the
> given model instance. Hence, the individual SQL queries for each column.
>
> To solve this, we would need to change DeferredAttribute to find all
> *other* DeferredAttributes on the given model and load them in a single
> query somehow.
>
> Also, I should mention that this should be *optional* behavior, as the
> current behavior is reasonable for the common case. The API for specifying
> this "load everything" behavior is a separate discussion. Perhaps a keyword
> argument like: User.objects.only('username', loadall=True).
>
> Thoughts?
>
> Adrian
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-developers+unsubscr...@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.
> Visit this group at http://groups.google.com/group/django-developers?hl=en
> .
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
--
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote:
>
> Also, I should mention that this should be *optional* behavior, as the
> current behavior is reasonable for the common case. The API for specifying
> this "load everything" behavior is a separate discussion. Perhaps a keyword
> argument like: User.objects.only('username', loadall=True).
>
I could imagine a Meta attribute which introduces so called "deferred
groups" like SA has
http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-column-loading,
accessing one column of a group will load all columns of the group. Not
sure if we want that level of control, but I thought it would be worth to
look what SA can do in this regard.
Regards,
Florian
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Should saving a model with foreign keys referencing unsaved models raise an error?
On 25 avr. 2013, at 04:22, Yo-Yo Ma wrote: > The following example can throw a wrench in things, if you don't catch it > right away, since it fails silently. > > >>> instance.some_fk_field = unsaved_instance > >>> instance.save() > > The following example bit somebody I worked with a couple years back as well. > > >>> instance.some_m2m.add(unsaved_instance) > > Would it suffice to modify the ORM to fail loudly when either of the above > examples occur, or do they both represent documented features (thus > preventing them from being "fixed")? Yes, that should fail noisily. I may have fixed a few cases recently, but not all of them. There's at least one ticket in Trac about this, and it's probably assigned to me. -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On Thu, Apr 25, 2013 at 2:10 PM, Florian Apolloner wrote:
> On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote:
>>
>> Also, I should mention that this should be *optional* behavior, as the
>> current behavior is reasonable for the common case. The API for specifying
>> this "load everything" behavior is a separate discussion. Perhaps a keyword
>> argument like: User.objects.only('username', loadall=True).
>>
>
> I could imagine a Meta attribute which introduces so called "deferred
> groups" like SA has
> http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-column-loading,
> accessing one column of a group will load all columns of the group. Not
> sure if we want that level of control, but I thought it would be worth to
> look what SA can do in this regard.
>
I think groups are a very good abstraction for this problem. The two common
cases are probably "Load this column alone, because it's potentially a big
honking blob of text or binary" and "Load everything we don't have on this
object, because we are actually using it actively". Groups let you solve
both problems flexibly. The downside is that they might not be very DRY,
having to repeat group="everything" over and over if you just want to load
it all on first access.
Best,
Alex Ogier
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On 25 huhti, 23:44, Alex Ogier wrote:
> On Thu, Apr 25, 2013 at 2:10 PM, Florian Apolloner
> wrote:
>
> > On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote:
>
> >> Also, I should mention that this should be *optional* behavior, as the
> >> current behavior is reasonable for the common case. The API for specifying
> >> this "load everything" behavior is a separate discussion. Perhaps a keyword
> >> argument like: User.objects.only('username', loadall=True).
>
> > I could imagine a Meta attribute which introduces so called "deferred
> > groups" like SA has
> >http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-...,
> > accessing one column of a group will load all columns of the group. Not
> > sure if we want that level of control, but I thought it would be worth to
> > look what SA can do in this regard.
>
> I think groups are a very good abstraction for this problem. The two common
> cases are probably "Load this column alone, because it's potentially a big
> honking blob of text or binary" and "Load everything we don't have on this
> object, because we are actually using it actively". Groups let you solve
> both problems flexibly. The downside is that they might not be very DRY,
> having to repeat group="everything" over and over if you just want to load
> it all on first access.
+1 to this approach.
IMO load everything is a better default than one field at a time. When
deferred field loading happens something has already gone wrong. In
almost every case it is better to try to minimize the amount of
queries than the amount of loaded fields. The cases where you have
deferred multiple fields, need only one of them later on, and there is
a field that you can't load due to potentially huge value are
hopefully rare.
But, I don't really care too much. If the objects come from DB queries
instead of something like the use case of Adrian then if you end up
doing deferred field loading you have already failed. So, even an
error on deferred field access would work for my use cases of defer...
- Anssi
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On 25 huhti, 20:08, Alex Gaynor wrote: > This sounds like a reasonable request, I don't yet have an opinion on API > or anything. One tiny thing I'd like to note though, "change DeferredAttribute > to find all *other* DeferredAttributes". I don't think `finding` is the > right way to think about it, a `DeferredAttribute` with loadall semantics > should know about the specific set of objects that came from the queryset. Searching for deferred fields already happens in some places (at least model.__init__ and model.save()). It would be possible to store the set of deferred fields in model._state on load from DB, but why record this information (with real possibility of messing it up if users do something like direct __dict__ assignments after load from DB) when there is a way to calculate the set of deferred fields on demand? One possible reason is to avoid the expense of the calculation, but the calculation costs very little compared to the DB query about to happen. I actually tried to store the set of deferred fields per instance when polishing the update_fields patch but the approach turned out to be uglier at least for the update_fields case. Now, if there would be some way to avoid the dynamic model subclassing used by deferred loading then storing the set of deferred fields per instance would be more than welcome. One possibility might be defining Model.__getattr__. If you end up in __getattr__ and the value isn't present in the model's __dict__ but the attname is present in the model._state.deferred_fields, then you know to load it from DB. I assume there are some reasons why this wasn't done in the first place... Maybe descriptors for custom fields would break or the above mentioned direct __dict__ access case would fail? - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
Sorry, I misunderstood the original request. Yes, you're right Anssi and Adrian, finding them on demand is reasonable. Alex On Thu, Apr 25, 2013 at 4:59 PM, Anssi Kääriäinen wrote: > On 25 huhti, 20:08, Alex Gaynor wrote: > > This sounds like a reasonable request, I don't yet have an opinion on API > > or anything. One tiny thing I'd like to note though, "change > DeferredAttribute > > to find all *other* DeferredAttributes". I don't think `finding` is the > > right way to think about it, a `DeferredAttribute` with loadall semantics > > should know about the specific set of objects that came from the > queryset. > > Searching for deferred fields already happens in some places (at least > model.__init__ and model.save()). It would be possible to store the > set of deferred fields in model._state on load from DB, but why record > this information (with real possibility of messing it up if users do > something like direct __dict__ assignments after load from DB) when > there is a way to calculate the set of deferred fields on demand? One > possible reason is to avoid the expense of the calculation, but the > calculation costs very little compared to the DB query about to > happen. I actually tried to store the set of deferred fields per > instance when polishing the update_fields patch but the approach > turned out to be uglier at least for the update_fields case. > > Now, if there would be some way to avoid the dynamic model subclassing > used by deferred loading then storing the set of deferred fields per > instance would be more than welcome. One possibility might be defining > Model.__getattr__. If you end up in __getattr__ and the value isn't > present in the model's __dict__ but the attname is present in the > model._state.deferred_fields, then you know to load it from DB. I > assume there are some reasons why this wasn't done in the first > place... Maybe descriptors for custom fields would break or the above > mentioned direct __dict__ access case would fail? > > - Anssi > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers?hl=en > . > For more options, visit https://groups.google.com/groups/opt_out. > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [#20291] Add method to reload `AppCache`
On Thu, Apr 25, 2013 at 8:50 PM, wrote: > Hi, > > Speaking of: > https://code.djangoproject.com/ticket/20291 > > I don't see how is this ticket related to #3591 (which BTW says "add > support for custom app_label and verbose_name"). This one is a little > atomic piece of work and a handy addition that I need as well, and waiting > for a ticket that is already six years old to be closed some day means that > it may happen when I'm retired. So, what do you think about reopening this > one? > > I'll confirm that #20291 is a duplicate of #3591. However, I'll also agree that the connection isn't obvious. You need to know this history, and see the big picture. #3591 has become, over time, the holding ticket for a bigger issue known as the "app refactor". The underlying issue is that Django doesn't currently have any place to store app-related configuration, and doesn't have any guaranteed order for performing that app-related configuration. As a related issue, this means that the app cache needs to be refactored to allow for reliable ordering. Bringing it back to #20291, in order to test these new capabilities means adding features to clean up and reset the app cache. #3591 is in a similar position to that of #3011 (custom auth.User) at about this time last year -- that is, we know what the problem is, and there's someone working on it. Preston Holmes had a working patch at DjangoCon US last year; however, he's recently had a revelation on how to significantly simplify some of the internals, which he's working on at the moment. I'm sure he'll call for comments and reviews when the code is in an appropriate state. Just because it's old doesn't mean it's being ignored, and integrating a short term workaround would introduce constraints and expectations on how the new API would work. To that end, I'm in agreement with Jacob and Augustin on #20291 -- this is a duplicate of #3591, and I'd rather see effort concentrated on fixing that ticket. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
