[#20291] Add method to reload `AppCache`
Hi, Speaking of: https://code.djangoproject.com/ticket/20291 I don't see how is this ticket related to #3591 (which BTW says "add support for custom app_label and verbose_name"). This one is a little atomic piece of work and a handy addition that I need as well, and waiting for a ticket that is already six years old to be closed some day means that it may happen when I'm retired. So, what do you think about reopening this one? Greetings, Robert -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Changing deferred model attribute behavior
At the moment, if you call defer() or only() on a QuerySet, then access the deferred fields individually, *each* access of a deferred field will result in a separate query. For example, assuming a User model with username/bio/location fields, this is what currently happens: """ >>> u = User.objects.only('username').get(id=1) # Results in: SELECT id, username FROM users WHERE id=1 >>> u.bio # Results in SELECT bio FROM users WHERE id=1 >>> u.location # Results in SELECT location FROM users WHERE id=1 """ I'd like there to be a way of retrieving *all* deferred fields the first time a deferred field is accessed. So with the above example, this would happen instead: """ >>> u = User.objects.only('username').get(id=1) # Results in: SELECT id, username FROM users WHERE id=1 >>> u.bio # Results in SELECT bio, location FROM users WHERE id=1 >>> u.location # No database query """ Here's my use case. In my app, I'm storing frequently accessed user attributes (id, username, etc.) in signed cookies, to prevent an unnecessary database hit on every page view. I create a User object with those attributes, so that I can take advantage of various methods on the User class, but I want the other User fields to be lazily loaded. I have this all working except for the last piece, which is to change lazy loading such that it loads *everything* else the first time a deferred field is accessed. So that's the "what" and "why" -- here's the how... The current implementation is in django/db/models/query_utils.py -- see the DeferredAttribute class. When you call only() or defer() on a QuerySet, the resulting model instance will have a DeferredAttribute instance for each deferred field. The problem is that DeferredAttributes only load their own column's data, not the data for all *other* DeferredAttributes on the given model instance. Hence, the individual SQL queries for each column. To solve this, we would need to change DeferredAttribute to find all *other* DeferredAttributes on the given model and load them in a single query somehow. Also, I should mention that this should be *optional* behavior, as the current behavior is reasonable for the common case. The API for specifying this "load everything" behavior is a separate discussion. Perhaps a keyword argument like: User.objects.only('username', loadall=True). Thoughts? Adrian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
This sounds like a reasonable request, I don't yet have an opinion on API or anything. One tiny thing I'd like to note though, "change DeferredAttribute to find all *other* DeferredAttributes". I don't think `finding` is the right way to think about it, a `DeferredAttribute` with loadall semantics should know about the specific set of objects that came from the queryset. Alex On Thu, Apr 25, 2013 at 12:06 PM, Adrian Holovaty wrote: > At the moment, if you call defer() or only() on a QuerySet, then access > the deferred fields individually, *each* access of a deferred field will > result in a separate query. > > For example, assuming a User model with username/bio/location fields, this > is what currently happens: > > """ > >>> u = User.objects.only('username').get(id=1) > # Results in: SELECT id, username FROM users WHERE id=1 > > >>> u.bio > # Results in SELECT bio FROM users WHERE id=1 > > >>> u.location > # Results in SELECT location FROM users WHERE id=1 > """ > > I'd like there to be a way of retrieving *all* deferred fields the first > time a deferred field is accessed. So with the above example, this would > happen instead: > > """ > >>> u = User.objects.only('username').get(id=1) > # Results in: SELECT id, username FROM users WHERE id=1 > > >>> u.bio > # Results in SELECT bio, location FROM users WHERE id=1 > > >>> u.location > # No database query > """ > > Here's my use case. In my app, I'm storing frequently accessed user > attributes (id, username, etc.) in signed cookies, to prevent an > unnecessary database hit on every page view. I create a User object with > those attributes, so that I can take advantage of various methods on the > User class, but I want the other User fields to be lazily loaded. I have > this all working except for the last piece, which is to change lazy loading > such that it loads *everything* else the first time a deferred field is > accessed. > > So that's the "what" and "why" -- here's the how... > > The current implementation is in django/db/models/query_utils.py -- see > the DeferredAttribute class. When you call only() or defer() on a QuerySet, > the resulting model instance will have a DeferredAttribute instance for > each deferred field. The problem is that DeferredAttributes only load their > own column's data, not the data for all *other* DeferredAttributes on the > given model instance. Hence, the individual SQL queries for each column. > > To solve this, we would need to change DeferredAttribute to find all > *other* DeferredAttributes on the given model and load them in a single > query somehow. > > Also, I should mention that this should be *optional* behavior, as the > current behavior is reasonable for the common case. The API for specifying > this "load everything" behavior is a separate discussion. Perhaps a keyword > argument like: User.objects.only('username', loadall=True). > > Thoughts? > > Adrian > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers?hl=en > . > For more options, visit https://groups.google.com/groups/opt_out. > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote: > > Also, I should mention that this should be *optional* behavior, as the > current behavior is reasonable for the common case. The API for specifying > this "load everything" behavior is a separate discussion. Perhaps a keyword > argument like: User.objects.only('username', loadall=True). > I could imagine a Meta attribute which introduces so called "deferred groups" like SA has http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-column-loading, accessing one column of a group will load all columns of the group. Not sure if we want that level of control, but I thought it would be worth to look what SA can do in this regard. Regards, Florian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Should saving a model with foreign keys referencing unsaved models raise an error?
On 25 avr. 2013, at 04:22, Yo-Yo Ma wrote: > The following example can throw a wrench in things, if you don't catch it > right away, since it fails silently. > > >>> instance.some_fk_field = unsaved_instance > >>> instance.save() > > The following example bit somebody I worked with a couple years back as well. > > >>> instance.some_m2m.add(unsaved_instance) > > Would it suffice to modify the ORM to fail loudly when either of the above > examples occur, or do they both represent documented features (thus > preventing them from being "fixed")? Yes, that should fail noisily. I may have fixed a few cases recently, but not all of them. There's at least one ticket in Trac about this, and it's probably assigned to me. -- Aymeric. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On Thu, Apr 25, 2013 at 2:10 PM, Florian Apolloner wrote: > On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote: >> >> Also, I should mention that this should be *optional* behavior, as the >> current behavior is reasonable for the common case. The API for specifying >> this "load everything" behavior is a separate discussion. Perhaps a keyword >> argument like: User.objects.only('username', loadall=True). >> > > I could imagine a Meta attribute which introduces so called "deferred > groups" like SA has > http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-column-loading, > accessing one column of a group will load all columns of the group. Not > sure if we want that level of control, but I thought it would be worth to > look what SA can do in this regard. > I think groups are a very good abstraction for this problem. The two common cases are probably "Load this column alone, because it's potentially a big honking blob of text or binary" and "Load everything we don't have on this object, because we are actually using it actively". Groups let you solve both problems flexibly. The downside is that they might not be very DRY, having to repeat group="everything" over and over if you just want to load it all on first access. Best, Alex Ogier -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On 25 huhti, 23:44, Alex Ogier wrote: > On Thu, Apr 25, 2013 at 2:10 PM, Florian Apolloner > wrote: > > > On Thursday, April 25, 2013 7:06:06 PM UTC+2, Adrian Holovaty wrote: > > >> Also, I should mention that this should be *optional* behavior, as the > >> current behavior is reasonable for the common case. The API for specifying > >> this "load everything" behavior is a separate discussion. Perhaps a keyword > >> argument like: User.objects.only('username', loadall=True). > > > I could imagine a Meta attribute which introduces so called "deferred > > groups" like SA has > >http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#deferred-..., > > accessing one column of a group will load all columns of the group. Not > > sure if we want that level of control, but I thought it would be worth to > > look what SA can do in this regard. > > I think groups are a very good abstraction for this problem. The two common > cases are probably "Load this column alone, because it's potentially a big > honking blob of text or binary" and "Load everything we don't have on this > object, because we are actually using it actively". Groups let you solve > both problems flexibly. The downside is that they might not be very DRY, > having to repeat group="everything" over and over if you just want to load > it all on first access. +1 to this approach. IMO load everything is a better default than one field at a time. When deferred field loading happens something has already gone wrong. In almost every case it is better to try to minimize the amount of queries than the amount of loaded fields. The cases where you have deferred multiple fields, need only one of them later on, and there is a field that you can't load due to potentially huge value are hopefully rare. But, I don't really care too much. If the objects come from DB queries instead of something like the use case of Adrian then if you end up doing deferred field loading you have already failed. So, even an error on deferred field access would work for my use cases of defer... - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
On 25 huhti, 20:08, Alex Gaynor wrote: > This sounds like a reasonable request, I don't yet have an opinion on API > or anything. One tiny thing I'd like to note though, "change DeferredAttribute > to find all *other* DeferredAttributes". I don't think `finding` is the > right way to think about it, a `DeferredAttribute` with loadall semantics > should know about the specific set of objects that came from the queryset. Searching for deferred fields already happens in some places (at least model.__init__ and model.save()). It would be possible to store the set of deferred fields in model._state on load from DB, but why record this information (with real possibility of messing it up if users do something like direct __dict__ assignments after load from DB) when there is a way to calculate the set of deferred fields on demand? One possible reason is to avoid the expense of the calculation, but the calculation costs very little compared to the DB query about to happen. I actually tried to store the set of deferred fields per instance when polishing the update_fields patch but the approach turned out to be uglier at least for the update_fields case. Now, if there would be some way to avoid the dynamic model subclassing used by deferred loading then storing the set of deferred fields per instance would be more than welcome. One possibility might be defining Model.__getattr__. If you end up in __getattr__ and the value isn't present in the model's __dict__ but the attname is present in the model._state.deferred_fields, then you know to load it from DB. I assume there are some reasons why this wasn't done in the first place... Maybe descriptors for custom fields would break or the above mentioned direct __dict__ access case would fail? - Anssi -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: Changing deferred model attribute behavior
Sorry, I misunderstood the original request. Yes, you're right Anssi and Adrian, finding them on demand is reasonable. Alex On Thu, Apr 25, 2013 at 4:59 PM, Anssi Kääriäinen wrote: > On 25 huhti, 20:08, Alex Gaynor wrote: > > This sounds like a reasonable request, I don't yet have an opinion on API > > or anything. One tiny thing I'd like to note though, "change > DeferredAttribute > > to find all *other* DeferredAttributes". I don't think `finding` is the > > right way to think about it, a `DeferredAttribute` with loadall semantics > > should know about the specific set of objects that came from the > queryset. > > Searching for deferred fields already happens in some places (at least > model.__init__ and model.save()). It would be possible to store the > set of deferred fields in model._state on load from DB, but why record > this information (with real possibility of messing it up if users do > something like direct __dict__ assignments after load from DB) when > there is a way to calculate the set of deferred fields on demand? One > possible reason is to avoid the expense of the calculation, but the > calculation costs very little compared to the DB query about to > happen. I actually tried to store the set of deferred fields per > instance when polishing the update_fields patch but the approach > turned out to be uglier at least for the update_fields case. > > Now, if there would be some way to avoid the dynamic model subclassing > used by deferred loading then storing the set of deferred fields per > instance would be more than welcome. One possibility might be defining > Model.__getattr__. If you end up in __getattr__ and the value isn't > present in the model's __dict__ but the attname is present in the > model._state.deferred_fields, then you know to load it from DB. I > assume there are some reasons why this wasn't done in the first > place... Maybe descriptors for custom fields would break or the above > mentioned direct __dict__ access case would fail? > > - Anssi > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To post to this group, send email to django-developers@googlegroups.com. > Visit this group at http://groups.google.com/group/django-developers?hl=en > . > For more options, visit https://groups.google.com/groups/opt_out. > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: [#20291] Add method to reload `AppCache`
On Thu, Apr 25, 2013 at 8:50 PM, wrote: > Hi, > > Speaking of: > https://code.djangoproject.com/ticket/20291 > > I don't see how is this ticket related to #3591 (which BTW says "add > support for custom app_label and verbose_name"). This one is a little > atomic piece of work and a handy addition that I need as well, and waiting > for a ticket that is already six years old to be closed some day means that > it may happen when I'm retired. So, what do you think about reopening this > one? > > I'll confirm that #20291 is a duplicate of #3591. However, I'll also agree that the connection isn't obvious. You need to know this history, and see the big picture. #3591 has become, over time, the holding ticket for a bigger issue known as the "app refactor". The underlying issue is that Django doesn't currently have any place to store app-related configuration, and doesn't have any guaranteed order for performing that app-related configuration. As a related issue, this means that the app cache needs to be refactored to allow for reliable ordering. Bringing it back to #20291, in order to test these new capabilities means adding features to clean up and reset the app cache. #3591 is in a similar position to that of #3011 (custom auth.User) at about this time last year -- that is, we know what the problem is, and there's someone working on it. Preston Holmes had a working patch at DjangoCon US last year; however, he's recently had a revelation on how to significantly simplify some of the internals, which he's working on at the moment. I'm sure he'll call for comments and reviews when the code is in an appropriate state. Just because it's old doesn't mean it's being ignored, and integrating a short term workaround would introduce constraints and expectations on how the new API would work. To that end, I'm in agreement with Jacob and Augustin on #20291 -- this is a duplicate of #3591, and I'd rather see effort concentrated on fixing that ticket. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers?hl=en. For more options, visit https://groups.google.com/groups/opt_out.