Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Eugene Lazutkin wrote: >Woot! 1539 and 1442 are both committed! Now all multithreaded guys, and >MySQL users can start up projects right out of the box without digging >for "magic incantations" in tickets, mail lists, and blogs! > > Woots and thanks to Adrian and Eugene from me too :-) I hope the next step would be a file upload fix (http://code.djangoproject.com/ticket/1484). I think it's one of the things that can bite an unsuspecting programmer when after rolling a site on production server it dies under some non-toy load. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Adrian Holovaty wrote: > > Thanks, guys! I've commited that. Woot! 1539 and 1442 are both committed! Now all multithreaded guys, and MySQL users can start up projects right out of the box without digging for "magic incantations" in tickets, mail lists, and blogs! Thanks, Eugene --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
On 3/25/06, Eugene Lazutkin <[EMAIL PROTECTED]> wrote: > The patch is in the Trac: http://code.djangoproject.com/ticket/1539. > I've submitted both versions for the Django trunk and the magic-removal > branch. > > Big thanks to Alex Brown for finding the bug and testing the fix. Thanks, guys! I've commited that. Adrian -- Adrian Holovaty holovaty.com | djangoproject.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
The patch is in the Trac: http://code.djangoproject.com/ticket/1539. I've submitted both versions for the Django trunk and the magic-removal branch. Big thanks to Alex Brown for finding the bug and testing the fix. Thanks, Eugene Alex Brown wrote: > Ivan Sagalaev wrote: >> I think so too... I was investigating Alex's report and it looks like >> somewhere onr thread is creating a QueryDict and another tries to access >> it while it's being created. It's the only place where it doesn't >> contain self._mutable. >> >> This may mean that mod_python somehow supplies shared request object >> which it shouldn't. On a mod-python list I saw a suggestion >> (http://www.modpython.org/pipermail/mod_python/2003-October/014398.html): >> >>> It means your Python that mod_python was built against doesn't support >>> threads. You might have a thread.py module under your python libs dir >>> but that doesn't mean your python supports threads. >>> >>> Go the the mod_python page, get the appropriate version per your apache >>> version, and then get EXACTLY the version of Python recommended. Build >>> this python in your source tree (with threads!) and mod_python against >>> this. Everything will work fine. >>> >> But I didn't put it here since it's to unfounded :-). Alex, could you >> check these versioning issues anyway? > > Ivan, > > My python and mod_python were both compiled with threading. Whilst > investigating these issues I did find mention in a few posts that > mod_python 3.1.4 and earlier having some thread related bugs. I was > running 3.1.3 and when I upgraded to 3.2.5b at least one of my apache > crashing issues seemed to disappeared. > > Eugene's patch did also seem to resolve the QueryDict issue I reported. > > Thanks > > Alex > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Ivan Sagalaev wrote: > I think so too... I was investigating Alex's report and it looks like > somewhere onr thread is creating a QueryDict and another tries to access > it while it's being created. It's the only place where it doesn't > contain self._mutable. > > This may mean that mod_python somehow supplies shared request object > which it shouldn't. On a mod-python list I saw a suggestion > (http://www.modpython.org/pipermail/mod_python/2003-October/014398.html): > > >It means your Python that mod_python was built against doesn't support > >threads. You might have a thread.py module under your python libs dir > >but that doesn't mean your python supports threads. > > > >Go the the mod_python page, get the appropriate version per your apache > >version, and then get EXACTLY the version of Python recommended. Build > >this python in your source tree (with threads!) and mod_python against > >this. Everything will work fine. > > > But I didn't put it here since it's to unfounded :-). Alex, could you > check these versioning issues anyway? Ivan, My python and mod_python were both compiled with threading. Whilst investigating these issues I did find mention in a few posts that mod_python 3.1.4 and earlier having some thread related bugs. I was running 3.1.3 and when I upgraded to 3.2.5b at least one of my apache crashing issues seemed to disappeared. Eugene's patch did also seem to resolve the QueryDict issue I reported. Thanks Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Ivan Sagalaev wrote: >> > I think so too... I was investigating Alex's report and it looks like > somewhere onr thread is creating a QueryDict and another tries to access > it while it's being created. It's the only place where it doesn't > contain self._mutable. > > This may mean that mod_python somehow supplies shared request object > which it shouldn't. On a mod-python list I saw a suggestion I reproduced it in python interpreter. I don't think it is mod_python-related problem. There are two problems as far as I can tell: 1) QueryDict uses the hacks with class-level variables, which are reassigned temporarily during copy(). Its base class (MultiValueDict) does the same. It's a no-no for multithreaded programming. The same goes for module-level variables. In short: all changes to common variables should be analyzed and synchronized, or avoided. The latter is the best approach. 2) copy.deepcopy() doesn't like to be fooled with dynamic class modifications, e.g. with replacing __setitem__. Example: = class Ex(dict): def __init__(self, mutable=True): dict.__init__(self) self._mutable = mutable def _setitem_if_mutable(self, key, value): print 'Setting', key, 'to', value self._assert_mutable() dict.__setitem__(self, key, value) __setitem__ = _setitem_if_mutable def _assert_mutable(self): if not self._mutable: raise AttributeError, "This Ex instance is immutable" def copy(self): "Returns a mutable copy of this object." import copy # Our custom __setitem__ must be disabled for copying machinery. old_mutable = self._mutable self._mutable = True cp = copy.deepcopy(self) self._mutable = old_mutable return cp a = Ex() # it works fine # a is fully functional b = a.copy() # it works fine # b is fully functional c = b.copy() # it blows: AttributeError: 'Ex' object has no attribute '_mutable' c = a.copy() # it works fine = Try to play with it. The right way to handle copy/deepcopy is to define proper ___copy__/__deepcopy__ methods, or to define custom pickling helpers. Incidentally it was a point of the hacks in #1. I implemented a patch for #1 and #2, and sent it to Alex. If he confirms it, I'll create a ticket. Otherwise I'll dig more. I am afraid we'll find more places with class-level (like this) or module-level (like in uncommitted #1442 patch) variables. Thanks, Eugene --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Eugene Lazutkin wrote: >Alex: I'll look into this problem. Most probably there is a racing >condition in multithreaded Django. It doesn't look like it is related to >database backends. > I think so too... I was investigating Alex's report and it looks like somewhere onr thread is creating a QueryDict and another tries to access it while it's being created. It's the only place where it doesn't contain self._mutable. This may mean that mod_python somehow supplies shared request object which it shouldn't. On a mod-python list I saw a suggestion (http://www.modpython.org/pipermail/mod_python/2003-October/014398.html): >It means your Python that mod_python was built against doesn't support >threads. You might have a thread.py module under your python libs dir >but that doesn't mean your python supports threads. > >Go the the mod_python page, get the appropriate version per your apache >version, and then get EXACTLY the version of Python recommended. Build >this python in your source tree (with threads!) and mod_python against >this. Everything will work fine. > But I didn't put it here since it's to unfounded :-). Alex, could you check these versioning issues anyway? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Alex Brown wrote: > > Let me know if there is anything I can do to help. In e-mail. Thanks, Eugene --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Eugene Lazutkin wrote: > Alex Brown wrote: > > Eugene Lazutkin wrote: > > > >> Alex Brown wrote: > >>> I am running django on a Windows server (apache 2.0.54/mod_python > >>> 3.1.3/postgres 8.0.3) > >>> > >>> Django would crash apache when running multiple automated scripts > >>> POSTing data at a high rate through a generic create_object view. > >>> > >>> After scanning the net I found various threading fix patches, and I > >>> ended up finding that the thread-local-storage patch seemed to stop the > >>> server from crashing. The problem I have now is that occasionally (1 in > >> Could you give a link to the patch? > >> > >> Thanks, > >> > >> Eugene > > > > Sure.. the patch I used was > > http://code.djangoproject.com/attachment/ticket/1442/magic.threading.2.patch > > > > Which is really very simple (VERY simple for postgres) > > OMG! Adrian: when is it going to be merged to the trunk and to the > magic-removal branch? Is there something wrong with it? Any second > thoughts? Hidden problems? Negative experience with the patch? It is > amazing that such important defect was patched 7 month ago (#463 for > MySQL) but is not accepted even now!!! Why??? > > Alex: I'll look into this problem. Most probably there is a racing > condition in multithreaded Django. It doesn't look like it is related to > database backends. The chances are there are more static objects, which > are (improperly) shared across threads. > > Thanks, > > Eugene Eugene, Let me know if there is anything I can do to help. Thanks, Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Alex Brown wrote: > Eugene Lazutkin wrote: > >> Alex Brown wrote: >>> I am running django on a Windows server (apache 2.0.54/mod_python >>> 3.1.3/postgres 8.0.3) >>> >>> Django would crash apache when running multiple automated scripts >>> POSTing data at a high rate through a generic create_object view. >>> >>> After scanning the net I found various threading fix patches, and I >>> ended up finding that the thread-local-storage patch seemed to stop the >>> server from crashing. The problem I have now is that occasionally (1 in >> Could you give a link to the patch? >> >> Thanks, >> >> Eugene > > Sure.. the patch I used was > http://code.djangoproject.com/attachment/ticket/1442/magic.threading.2.patch > > Which is really very simple (VERY simple for postgres) OMG! Adrian: when is it going to be merged to the trunk and to the magic-removal branch? Is there something wrong with it? Any second thoughts? Hidden problems? Negative experience with the patch? It is amazing that such important defect was patched 7 month ago (#463 for MySQL) but is not accepted even now!!! Why??? Alex: I'll look into this problem. Most probably there is a racing condition in multithreaded Django. It doesn't look like it is related to database backends. The chances are there are more static objects, which are (improperly) shared across threads. Thanks, Eugene --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Eugene Lazutkin wrote: > Alex Brown wrote: > > I am running django on a Windows server (apache 2.0.54/mod_python > > 3.1.3/postgres 8.0.3) > > > > Django would crash apache when running multiple automated scripts > > POSTing data at a high rate through a generic create_object view. > > > > After scanning the net I found various threading fix patches, and I > > ended up finding that the thread-local-storage patch seemed to stop the > > server from crashing. The problem I have now is that occasionally (1 in > > Could you give a link to the patch? > > Thanks, > > Eugene Sure.. the patch I used was http://code.djangoproject.com/attachment/ticket/1442/magic.threading.2.patch Which is really very simple (VERY simple for postgres) Alex --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
Re: apache threading - 'QueryDict' object has no attribute '_mutable'
Alex Brown wrote: > I am running django on a Windows server (apache 2.0.54/mod_python > 3.1.3/postgres 8.0.3) > > Django would crash apache when running multiple automated scripts > POSTing data at a high rate through a generic create_object view. > > After scanning the net I found various threading fix patches, and I > ended up finding that the thread-local-storage patch seemed to stop the > server from crashing. The problem I have now is that occasionally (1 in Could you give a link to the patch? Thanks, Eugene --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
apache threading - 'QueryDict' object has no attribute '_mutable'
I am running django on a Windows server (apache 2.0.54/mod_python 3.1.3/postgres 8.0.3) Django would crash apache when running multiple automated scripts POSTing data at a high rate through a generic create_object view. After scanning the net I found various threading fix patches, and I ended up finding that the thread-local-storage patch seemed to stop the server from crashing. The problem I have now is that occasionally (1 in 20 POSTs), I get an AttributeError exception 'QueryDict' object has no attribute '_mutable' django\http\__init__.py in _assert_mutable, line 79. It's asserting during the request.POST.copy() in create_object (generic view) This output was from the magic removal branch, however I think the behaviour is the same when using the trunk with the thread-local-storage patch. I understand that prefork is the way to go to stop these issues, but I can't see an easy way to do that in Windows (which I can't move from). Has anyone else solved the threading issue on Windows ? Is the thread-local-storage patch not the way to go ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users -~--~~~~--~~--~--~---
