Re: queryset caching note in docs
On Fri, Oct 28, 2011 at 1:05 PM, Marco Paolini wrote: > it's a bit more complex: there are basically two phases: > 1) row fetching from db using cursor.fetchmany() > 2) model instance creation in queryset > > both are delayed as much as possible (queryset lazyness) > > phase two (model instance creation) depends on how you evaluate the > queryset: > - testing for truth will only create enough instances to fill the cache > - calling .exists() will try to fetch a single object and leave the cache > alone > - iterating will create instances in batches > - .iterator() will create the instances one at the time (but on the same db > cursos using cursor.fetchmany() > - many other options... > You are probably more informed about the internals than me, but isn't it irrelevant about when the instances are created; they are created from the same query, even if the instances are not instantiated until iterated to. Any changes to the database that happened after that query is executed will not be reflected in the instances that are created from that query. IE the critical point is when the query is executed, not when the data is fetched from the db cursor. Cheers Tom -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: queryset caching note in docs
On 28/10/2011 12:58, Tom Evans wrote: On Fri, Oct 28, 2011 at 11:23 AM, Marco Paolini wrote: Hi all, I'd like to add a small note of warning about queryset caching in dos, in topics/db/queries.txt "Caching and QuerySets" section, something like: Keep in mind when looping through a queryset and altering data that might be returned by the same queryset, that instances might not be "fresh". You might want to refresh the instance before calling some table-wide data-altering method on it:: >>> for obj in Entry.objects.all(): >>> obj = Entry.objects.get(pk=obj.pk) >>> obj.alter_some_other_entries() Isn't that self evident? If you generate a list of models, and then change the underlying table, why would they be fresh? It's also nothing to do with queryset caching, or even querysets. You may as well use this example: "for obj in qs" does not simply generate a list of instances, it iterates through qs and, at db API level it issues a fetchmany trying to hit database only when qs cache is empty. Of course the qs cache is a list of instances. obj1 = Entry.objects.get(id=i) obj2 = Entry.objects.get(id=j) obj1.alter_other_objects_in_table() # oh noes, obj2 is not what is in the database I'd go with something that explains that model instances are created from the queryset as soon as you evaluate it, not as you iterate through it, but I'm pretty sure that is already there. it's a bit more complex: there are basically two phases: 1) row fetching from db using cursor.fetchmany() 2) model instance creation in queryset both are delayed as much as possible (queryset lazyness) phase two (model instance creation) depends on how you evaluate the queryset: - testing for truth will only create enough instances to fill the cache - calling .exists() will try to fetch a single object and leave the cache alone - iterating will create instances in batches - .iterator() will create the instances one at the time (but on the same db cursos using cursor.fetchmany() - many other options... phase one can be executed in chunks, depending on db backend (sqlite no chunks, postgre chunks) Queryset "laziness" MIGHT lead to a WRONG assumption that objects in a for loop are fresh. The note only wanted to stress the fact that you'll get "stale" objects if you do table-wide altering operations inside the loop I agree it's not surprising and it's not going to make your day happier, but it might be interesting to someone. On the other hand, your objections make sense: it should be quite clear already that in the for loop you're fetching objects using a cursor and altering data inside the loop will not change objects returned by queryset. cheers, Marco -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: queryset caching note in docs
On Fri, Oct 28, 2011 at 11:23 AM, Marco Paolini wrote: > Hi all, > > I'd like to add a small note of warning about queryset caching in dos, > in topics/db/queries.txt "Caching and QuerySets" section, > something like: > > Keep in mind when looping through a queryset and altering data that > might be returned by the same queryset, that instances might not > be "fresh". > > You might want to refresh the instance before calling some table-wide > data-altering > method on it:: > > >>> for obj in Entry.objects.all(): > >>> obj = Entry.objects.get(pk=obj.pk) > >>> obj.alter_some_other_entries() > Isn't that self evident? If you generate a list of models, and then change the underlying table, why would they be fresh? It's also nothing to do with queryset caching, or even querysets. You may as well use this example: obj1 = Entry.objects.get(id=i) obj2 = Entry.objects.get(id=j) obj1.alter_other_objects_in_table() # oh noes, obj2 is not what is in the database I'd go with something that explains that model instances are created from the queryset as soon as you evaluate it, not as you iterate through it, but I'm pretty sure that is already there. Cheers Tom -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
queryset caching note in docs
Hi all, I'd like to add a small note of warning about queryset caching in dos, in topics/db/queries.txt "Caching and QuerySets" section, something like: Keep in mind when looping through a queryset and altering data that might be returned by the same queryset, that instances might not be "fresh". You might want to refresh the instance before calling some table-wide data-altering method on it:: >>> for obj in Entry.objects.all(): >>> obj = Entry.objects.get(pk=obj.pk) >>> obj.alter_some_other_entries() Should I open a ticket or maybe it's a too specific use-case? Cheers, Marco -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
