Re: Lazy-loading model fields
Hey Alex, On the trac ticket, I mentioned that my motivating reason for adding specifying default-lazy fields in the Field definition (i.e., without having to specify them on every query set) is the implicit queries that Django creates for foreign key lookups (e.g., item = obj.relateditem causes an implicit SQL query, and I want to avoid loading large text fields on the related item). Also, to a lesser extent, it would be good to avoid loading these columns in select_related(). I hadn't thought before about subclassing Query/QuerySet/Manager/ TextField to do this without changing the installed version of Django. That's an interesting idea. My change did also involve changing Options (i.e., _meta), though. Currently, however, the Query and QuerySet classes are functionally decomposed in a way that makes it hard to override most of this functionality in a subclass without copy-and-pasting a very large amount of code. (Compare, for instance, django.contrib.gis.db.models.sql.Query.get_columns with django.db.models.sql.Query.get_columns.) I just posted a patch to http://code.djangoproject.com/ticket/9368 which would make overriding this functionality somewhat easier. But it would still involve a considerable amount of code duplication if that logic wasn't in the Query base class. -Jesse On Oct 14, 10:00 am, "[EMAIL PROTECTED]" <[EMAIL PROTECTED]> wrote: > I think the sensible solution is to have what the ticket originally > suggested, a queryset/manager method to control whether a field is > lazy loading, and yes, that can be implemented without patching > django, you can, of course, make this the default by overiding the > default manager. > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
I think the sensible solution is to have what the ticket originally suggested, a queryset/manager method to control whether a field is lazy loading, and yes, that can be implemented without patching django, you can, of course, make this the default by overiding the default manager. On Oct 14, 12:56 pm, "Marty Alchin" <[EMAIL PROTECTED]> wrote: > On Tue, Oct 14, 2008 at 12:34 PM, Jesse Young <[EMAIL PROTECTED]> wrote: > > A LazyTextField that manages a dynamic model behind the scenes sounds > > like a neat idea. That would certainly make the Django side of this > > easier to manage. (But of course I have a slight bias towards code > > that exists already -- has anyone actually implemented this?) > > I don't think it's out there anywhere, but if you like I could whip up > the code for it in about 15 minutes (probably less). I just haven't > bothered because I don't personally need it. > > > Anyway, I'm only trying to point out that inline lazy fields are > > *sometimes* useful, and that Django shouldn't force people to do it by > > splitting out fields into separate tables. > > Damn it. Now you've gotten me thinking of ways to possibly manage a > field the way you want (one table, lazy loading, no data migration, no > patches to django, one-line change to the model). I'm 98% sure it's > possible, and I know of many of the things that would need to be > addressed to do so, but I really shouldn't be spending my time on it. > I probably won't be able to get it out of my head until I get it > working though, unfortunately. If I do write it down, I'll let you > know. > > Of course, it would also have to come with big biohazard signs warning > people that there could (and likely would) be unforeseen side-effects, > particularly in the admin, serializers, modelforms ... actually, > pretty much anything that introspects your model would never see that > field. I'm not sure if that's suitable for your case, but it certainly > wouldn't be good enough for wide distribution. > > -Gul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
On Tue, Oct 14, 2008 at 12:34 PM, Jesse Young <[EMAIL PROTECTED]> wrote: > A LazyTextField that manages a dynamic model behind the scenes sounds > like a neat idea. That would certainly make the Django side of this > easier to manage. (But of course I have a slight bias towards code > that exists already -- has anyone actually implemented this?) I don't think it's out there anywhere, but if you like I could whip up the code for it in about 15 minutes (probably less). I just haven't bothered because I don't personally need it. > Anyway, I'm only trying to point out that inline lazy fields are > *sometimes* useful, and that Django shouldn't force people to do it by > splitting out fields into separate tables. Damn it. Now you've gotten me thinking of ways to possibly manage a field the way you want (one table, lazy loading, no data migration, no patches to django, one-line change to the model). I'm 98% sure it's possible, and I know of many of the things that would need to be addressed to do so, but I really shouldn't be spending my time on it. I probably won't be able to get it out of my head until I get it working though, unfortunately. If I do write it down, I'll let you know. Of course, it would also have to come with big biohazard signs warning people that there could (and likely would) be unforeseen side-effects, particularly in the admin, serializers, modelforms ... actually, pretty much anything that introspects your model would never see that field. I'm not sure if that's suitable for your case, but it certainly wouldn't be good enough for wide distribution. -Gul --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
> And LazyTextField could create the associated model behind the scenes, > set up a descriptor for loading its contents only when necessary, > cache those contents so they only get queried once, etc. And all this > without a single patch to Django's core and without you having to > manually maintain another model. A LazyTextField that manages a dynamic model behind the scenes sounds like a neat idea. That would certainly make the Django side of this easier to manage. (But of course I have a slight bias towards code that exists already -- has anyone actually implemented this?) > Admittedly, the relational approach would require a data migration, > but that's life sometimes. When an application is live and has a very large amount of data and users, avoiding data migration is extremely important. Data migration means that when we update our live website, someone has to run sql commands on our live database servers, which takes time and can add a lot of load; they have to reason about what would happen if someone visits our website if the sql is updated but not the python code (or vice versa), or whether they have to disable the website while the migration occurs. And the person who updates the website might be a different person than the developer who made the code change. So avoiding data migration just by making a one-line change in the Python code is a huge plus. Anyway, I'm only trying to point out that inline lazy fields are *sometimes* useful, and that Django shouldn't force people to do it by splitting out fields into separate tables. -Jesse --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Lazy-loading model fields
I won't pretend to know the whole story here, but I will take issue with some of your points. I'll leave others to people who have history on the subject. On Tue, Oct 14, 2008 at 3:14 AM, Jesse Young <[EMAIL PROTECTED]> wrote: > * If you split them out, you have to manually create one model every > time you create the other model. If by "manually create one model" you mean "type out a brand new model definition in Python" you're flat-out wrong. I've done quite a bit of work with generating models dynamically[1] and what you're asking for could be made very simple using that approach. Rather than this: thesis = models.TextField(lazy=True) you could use this instead: thesis = lazy_utils.LazyTextField() And LazyTextField could create the associated model behind the scenes, set up a descriptor for loading its contents only when necessary, cache those contents so they only get queried once, etc. And all this without a single patch to Django's core and without you having to manually maintain another model. > * If a fraction of your queries actually need those large fields, then > you have to do a join, which can be expensive. It doesn't address this, of course, but like I said, I'll leave that to the professionals. > * If your data access pattern changes after your application has > actual data, it is much harder to migrate your data than to make a one > line change to the application code (i.e., by adding lazy=True to the > field definition). Admittedly, the relational approach would require a data migration, but that's life sometimes. There are plenty of tools now to make changes like this relatively easy to deal with, and I expect a custom LazyTextField could be written to give extra hints to any of them. > * Maintaining more tables is more work than maintaining fewer tables. Again, this depends on what you mean by "maintaining". At a raw database level, yes, that's true, but if you're talking about the Python level, you can make that extra table almost non-existent and work with it as a regular field. Anyway, I don't expect this information to solve your problem, but I hope I've cleared up a few inaccuracies in your justification. Maybe it'll help you with whatever solution you do come up with. -Gul [1] http://code.djangoproject.com/wiki/DynamicModels --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
