On Mon, 2008-09-29 at 10:37 -0700, bo wrote: > > This little issue is really hard to replicate .. i've yet to find out > how to programatically do it because it certainly revolves around a > threading, object caching, and the Yield in Query.py (iteritems). > > I just wanted to post this here to see if anyone, with more experience > then i, knows howto replicate this in a test case world. > > On to the description.... > > The set up:: Apache 2.2.6 + Linux + Threaded (_not_ forked) + > mod_python + Django 1.X > > Suppose i have a 2 level caching object, that basically overloads the > function to store/get the object(s) from a Memcached overlord cache > (say 60 second expire), and 'local' cache (with a 1-2 second expire > time) .. the basic function is to keep a highly requested object > _per_ HTTP request local in ram w/o having to go back to the Memcached > cache. > > because of various iterators basing themselves off of "yield" > statements in db/models/query.py. Should 2 threads access the same > Local RAM cache object and try to iterate (yes the READS from the > cache are read/write locked, but this issue appears after the read > lock is released and the object is begin used), the ususal "Value > Error: Generator already running" exception is thrown. > > File "mything/models.py", line 1072, in _set_data > for p in self.data: > File "/usr/lib/python2.5/site-packages/django/db/models/query.py", > line 179, in _result_iter > self._fill_cache() > File "/usr/lib/python2.5/site-packages/django/db/models/query.py", > line 612, in _fill_cache > self._result_cache.append(self._iter.next()) > ValueError: generator already executing > > So, i'm aware this may not be a bug. But my own ignorance for not > doing something right. > > This does not happen very often in the servers i am running (about 10 > times a day on a 100k+ Django views/per day) which is why its really > hard to track down
I don't understand from your description what you're actually doing, but it sounds a lot like you're trying to read from the same QuerySet in multiple threads whilst it's still retrieving results from the database cursor. Don't do that. Firstly, database cursor result sets aren't necesarily safe to be shared across threads. QuerySet and Query objects probably are once the result set is populated, since every non-trivial operation on them creates a copy and parallel iteration is supported, but that's more by accident than design, since it's not worth the extra overhead: if you want to share QuerySets via caching, they contain the results (the result_cache is already fully primed). Nothing in Django will cache a connection to the database or a cursor result set, so can you break down your problem a bit more to describe where the simultaneous access is happing. You say "the usual ValueError", but I have never seen that raised by anything in Django. So I'm wondering if you're doing something fairly unusual here. That particular block of code is *designed* to be used in parallel iterators in the same thread, so it's safe in that respect. But if you're sharing a partially-read database cursor iterator across multiple threads, it might be a case of it breaks you get to keep all the pieces. I can't see why that would be necessary (if you want to cache the results, just cache the queryset and it will cache the results; if you want to cache the query, cache queryset.query and it will just cache the query; both of those cases are designed in and documented). Regards, Malcolm --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~----------~----~----~----~------~----~------~--~---
