ATOMIC_REQUESTS Alternative - better for highly concurrent systems
Hi,
I want to offer something the the community of Django users.
If you like the safety net of having each request handled by a transaction,
but don't want unnecessary blocking on highly concurrent web applications,
then the following may be of interest.
The Django transaction documentation offers you two choices:
1. Turn on the database setting ATOMIC_REQUESTS, and the for those view
functions which don't need transaction protection, decorate these with
@transaction.non_atomic_requests
2. Don't turn ATOMIC_REQUESTS for your database, and instead decorate
those view functions for which you want transaction protection with
@transaction.atomic.
However, there are a couple of downsides to both of these approaches:
1. There may be lots of view functions to be so decorated.
2. Many view functions follow the pattern where they accept both GET and
POST requests. The GET request returns the form to be filled in, and the
POST request submits the filled in form for processing. You want the POST
request to be processed by a transaction, as there may be multple tables to
be updated, however, you don't want the GET request to have the transaction
overhead. Therefore decorating such view functions with non_atomic_requests
or atomic won't do what you want.
A solution:
I offer a middleware module which looks to see if the request is a
modifying one (that is, one of POST, PUT, DELETE or PATCH). If not, it does
not use a transaction for the request. If it is a modifying request it will
use a transaction, provided that the database ATOMIC_REQUESTS is not on
(don't want to double up), and that the view function is not decorated with
@transaction.non_atomic_requests.
The middleware has to overcome a limitation that it cannot simpy do
something like "with transaction.atomic():", because in the process_view
method, the middleware has to return control before the view function is
called. It also has to work where several databases may be involved, so it
creates a list of transaction.Atomic instances, one per configured
database, saves them on the request object, and calls their __enter__()
methods.
The process_response() method then has to invoke the __exit__() method of
these Atomic instances in the reverse order, and handle any exceptions
which may occur.
It also provides a process_exception() method, which invokes the __exit__()
method of the Atomic instances, also in reverse order, passing to it the
exception information, and handling any exceptions which may occur. For
each Atomic instance whose __enter__() method was called, it has to invoke
the corresponding __exit__() method.
I welcome any comments from the group on this piece of middleware.
- Stephen Brooks
File: atomicmodifyingrequests.py
from django.db import connections, transaction
import sys
class AtomicModifyingRequestsMiddleware(object):
'''This middleware puts django.db.transaction.Atomic contexts (one per
database
if the database does not have the ATOMIC_REQUESTS set) around
the view function if the request method is a modifying one and
the view function is not annotated with
@transaction.non_atomic_requests.
Author: Stephen Brooks
Minimum Django Version: 1.6
'''
def process_view(self, request, view_func, view_args, view_kwargs):
if request.method in ('POST', 'PUT', 'DELETE', 'PATCH'):
non_atomic_requests = getattr(view_func,
'_non_atomic_requests', set())
try:
for db in connections.all():
if (not db.settings_dict['ATOMIC_REQUESTS'] and
db.alias not in non_atomic_requests):
if not hasattr(request,
'_atomic_modifying_requests_middleware_atomic_contexts'):
request._atomic_modifying_requests_middleware_atomic_contexts = []
atm_ctxt = transaction.Atomic(db.alias, True)
request._atomic_modifying_requests_middleware_atomic_contexts.append(atm_ctxt)
atm_ctxt.__enter__()
except Exception as e:
self.process_exception(request, e)
raise
return None
def process_response(self, request, response):
if hasattr(request,
'_atomic_modifying_requests_middleware_atomic_contexts'):
exc_info = (None, None, None,)
exc = None
for atm_ctxt in
reversed(request._atomic_modifying_requests_middleware_atomic_contexts):
try:
atm_ctxt.__exit__(*exc_info)
except Exception as exc:
exc_info = sys.exc_info()
if exc:
raise exc
return response
def process_exception(self, request, exception):
if hasattr(request,
'_atomic_modifying_requests_middleware_atomic_contexts'):
exc_in
Django 1.5 cached response has an empty body
With the change to HttpResponse made in Django 1.5, I'm finding that in my code, which caches a generated response, results in an empty body when that page is requested a second time. The first time the page is requested, it is not in the cache, and the page is generated normally and added to the cache. A subsequent request for the same page finds the response in the cache and that response is returned, but with a content length of zero. The reason is that the HttpResponse in Django 1.5 *does not reset the content iterator* when the content is requested to be iterated over again (the next time the response content is required). The HttpResponse in this case was not created by passing an iterator to HttpResponse. It is just using a string. The problem is that the __iter__ method of HttpResponse contains the line: > if not hasattr(self, '_iterator'): > self._iterator = iter(self._container) This prevents the iterator being reset the next time it is required to iterate over the content. _container still has the original content, but __iter__ does not reset the iterator as _iterator exists as an attribute since the first time that response was returned. The cached response is returning a used iterator, which returns no content. I suspect this is a bug. Any thoughts? What about a work-around in the meantime? When I retrieve the response from the cache, I could do: > response._iterator = iter(response._container) This works, but makes my code dependent on the internals of the HttpResponse class, which isn't great. Any better ideas? Kind regards, Steve -- You received this message because you are subscribed to the Google Groups "Django users" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscr...@googlegroups.com. To post to this group, send email to django-users@googlegroups.com. Visit this group at http://groups.google.com/group/django-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Re: BigAutoField
Yes, I'll post it there. Thanks, Steve On Wednesday, 23 January 2013 13:40:44 UTC, psjinx wrote: > > Hey Steve, > > You should ask this questions in django-developers mailing list as > it's related to development of django itself. It will be best to hear > comments from Core Developers. > > -- > Pankaj Singh > http://about.me/psjinx > > > On Wed, Jan 23, 2013 at 6:58 PM, SteveB > > wrote: > > Can anybody provide an update on the request to define a BigAutoField in > > Django? > > We could really use this model field type without having to do > workarounds > > and customizations. > > Can any of the Django developers comment on when this will be released? > > > > Thanks, > > Steve > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Django users" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/django-users/-/etnu2n_Fc6wJ. > > To post to this group, send email to > > django...@googlegroups.com. > > > To unsubscribe from this group, send email to > > django-users...@googlegroups.com . > > For more options, visit this group at > > http://groups.google.com/group/django-users?hl=en. > -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/hE66I1__eLcJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
BigAutoField
Can anybody provide an update on the request to define a BigAutoField in Django? We could really use this model field type without having to do workarounds and customizations. Can any of the Django developers comment on when this will be released? Thanks, Steve -- You received this message because you are subscribed to the Google Groups "Django users" group. To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/etnu2n_Fc6wJ. To post to this group, send email to django-users@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: Is Django right for what I am trying to do
Hi Bradley, Just my 2c re feeling comfortable with how django does things:- A high level walkthrough of which bits of Django are used and how they fit together. 1. You define the URL's that you want to present via a web server in urls.py. 2. In the urls.py file you configure url's and map these to views (python functions) held in views.py 3. The views can take an HTTP request as a parameter along with the parameters from the context. 4. The views use the parameters to retrieve a model instance, or multiple model instances. 5. The view 'calls' a template with the model instances received by 4. 6. The template processor parses the template & processes the model instances. 7. The view returns the processed template to the browser, or can redirects to another URL. 8. If you want a quick CRUD set of forms for admin use only you can use the admin functionality to provide this - you don't have to though. I think it takes a bit of getting your head around initially but it's very flexible/powerful when you do. There are loads of other things that Django can do, including Form processing but you can code forms yourself if you prefer. HIH On Jul 8, 6:43 pm, Bradley Hintze wrote: > Hi all > > I did the tutorial and I've spent the last two days in the > documentation trying, and failing, to figure out how to tie my model > to a view and ultimately a template that is served. The documentation > seems to do a lot of things (like write html) automatically which is > not what I want (seehttp://docs.djangoproject.com/en/dev/topics/forms/). I > am not > interested in admin sites, they make things easier but I am interested > in learning the code (i.e. the HTML and how it communicates with > python) rather than, in my view, just blackbox automation. > > I hope this makes sense. Please tell me if it doesn't. I want to > simply upload a file and then process the file using code I already > have. I would like to write the HTML code myself rather then having > django automatically do it. This is because I want to learn how to > process forms and not have a black box do everything for me. Are there > methods to simply get post data from a form that I write rather the > django automatically creating it (an example maybe)? are models > necessary for what I'm explaining? Is django right for what I'm > explaining? > > -- > Bradley J. Hintze > Graduate Student > Duke University > School of Medicine > 801-712-8799 -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-us...@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: Hierarchical data containing different models
Worth a look at django-treebeard as a tree library. I'm doing some work with it at the moment and I'm very impressed with it so far. It's pretty easy to separate the different data and connect to a tree- node object. IMO the api for tree work is good and all of the SQL complexity is hidden. You can choose the storage method of the tree's [i.e. Adjacency Lists or Materialised Paths], with a common interface between the different methods - so in theory you can change them later. Steve On Feb 24, 5:08 pm, Peter Reimer wrote: > Hi folks, > > I'm looking for a solution/tip for the following problem: I need to > store hierarchical data, but, with two different kinds of objects > (groups and items). I googled around and found the often suggested > mptt package. It looks really great, but I'm not sure about the best > way to save objects of different models with it. > > One idea is, to build the hierarchical structure with one model and > mptt and in it, I define two ForeignKeys to the concrete data objects > I want to store (ForeignKey(Group) and Foreignkey(Item)). But this > sounds a bit strange to me. I think there should be a much smarter > way. > > Any ideas? Many thanks in advance. > Peter -- You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-us...@googlegroups.com. To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
Re: Dynamic queryset when instantiating a form with a ModelChoiceField within a formset
Hi, well I have a solution, but it's not ideal. It involves extending the BaseFormSet class, and passing an extra parameter into the subclass's __init__ method, and saving this value as an attribute to self. The subclass also had to override the _construct_form method, copying all of the code of the BaseFormSet's _construct_form method (which is not good from a maintenance point of view, and definitely violates the Do not Repeat Yourself (DRY) principle), and when it instantiates the form, it passes in this extra parameter for the form to correctly initialize the queryset for the ModelChoiceField. How could this be improved? I think a better solution would involve the Django development team considering extending the BaseFormSet class to handle this situation. It could include an optional parameter (e.g. 'formConfig') in the BaseFormSet __init__ method, which would be passed as a parameter to each form's instantiation. The Django forms.BaseForm.__init__ would have to accept (though would ignore) this argument. Regards, Steve On May 14, 12:45 pm, SteveB wrote: > Hi, > I have a form which has a ModelChoiceField, whose queryset has to be > set dynamically (when the form is instantiated). > I have read an earlier posting on how to do > this:http://groups.google.com/group/django-users/browse_thread/thread/847e... > > My problem, is that I don't directly instantiate this form. The form > is used in a formset, > so the actual instantiation is done within Django code. > The above mentioned posting specifies that some extra parameter be > added to the > form's __init__ function argument list, and that it can be used to > determine the queryset for the ModelChoiceField, but how can I do this > with a formset? > > Thanks, > Steve --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Dynamic queryset when instantiating a form with a ModelChoiceField within a formset
Hi, I have a form which has a ModelChoiceField, whose queryset has to be set dynamically (when the form is instantiated). I have read an earlier posting on how to do this: http://groups.google.com/group/django-users/browse_thread/thread/847ec83e26dd7846/f0b15945a01eafe2?lnk=gst&q=queryset+modelchoicefield#f0b15945a01eafe2 My problem, is that I don't directly instantiate this form. The form is used in a formset, so the actual instantiation is done within Django code. The above mentioned posting specifies that some extra parameter be added to the form's __init__ function argument list, and that it can be used to determine the queryset for the ModelChoiceField, but how can I do this with a formset? Thanks, Steve --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to django-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Re: Webservers, Django thread-safeness, etc.
Actually, that statement, "to the best of anyone's knowledge, Django does not have issues running in threaded servers, and if/when bugs are found which contradict that we fix them" is what I was looking for. I probably should have asked, "Is Django intended and designed to be thread safe?" Obviously any large piece of software is susceptible to the introduction of bugs, and of course authors of extensions and application (e.g. programmers like me) can introduce non-threadsafe code. The important bottom line implied is that any thread unsafeness in the Django infrastructure would only be due to a bug, so such problems are most likely in my own development code. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
Webservers, Django thread-safeness, etc.
Deploying Django behind Apache is great for real web apps, but I also use Django for Desktop apps that employ a web browser as the UI. In this situation, a distribution package that depends on Apache is not an option (too much for end users to manage), and the Django development server's weaknesses (especially the one request at a time limitation) disqualify it. I've had positive experiences with the standalone CherryPy webserver in other contexts, and I've seen it recommended for use with Django, BUT I'm having problems. Maybe once in every dozen page requests (each which might result in a dozen or more http requests) it appears that the CherryPy webserver loses the request, i.e. it never gets passed up the WSGI stack to Django. If I hit the browser's reload button, (usually) the resubmitted page request is handled OK. Although I can't prove it, I'm suspicious that the problem may be that Django is not thread safe. Can anyone state for a fact that it is safe to use Django with a multi-threaded webserver? If not, then multi- threaded webservers like the CherryPy server are not really candidates for use with Django. Is there a high quality, server that's light enough to be packaged with a desktop app or other turn-key app that has been vetted for use with Django? I've researched the python based servers I could find mentioned via Google search, but everything I've found is either unmaintained, multi-threaded, or severely crippled. Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django users" group. To post to this group, send email to django-users@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-users?hl=en -~--~~~~--~~--~--~---
