Database race conditions when using multiple processes
Hello,
I was recently using the get_or_create method and noticed that it added
an entry twice into my database in very rare cases, because it
complained later that "get() returned more than one ___ -- it returned
2!". This problem didn't occur when using the development server.
My question is: Is there a way to avoid these race conditions by e.g.
locking the database? How can I do that?
This is my setup:
- Django (SVN) running with "python manage.py runfcgi method=prefork"
(there are 6 processes running)
- Lighttpd
- SQLite
To prove the race condition, I used the following model:
class Test(models.Model):
str = models.CharField(blank=True, null=True, maxlength=100)
The relevant part of my urls.conf is:
# Testing
(r'^test/$', 'test.main'),
(r'^test/process/$', 'test.process'),
And here's the view:
--- CUT HERE ---
from django.http import HttpResponse
import models
from json import load
def main(request):
return HttpResponse("""
http://prototype.conio.net/dist/prototype-1.4.0.js";
type="text/javascript">
http://www.json.org/json.js";
type="text/javascript">
function send_request(callback, text)
{
new Ajax.Request('process/' /* URL */, {
postBody:'text='+encodeURIComponent(text),
onComplete:callback
});
}
function debug(str)
{
var date = new Date();
str = date.toTimeString().substr(0, 8)+' '+str+'
';
$('debug').innerHTML = str+$('debug').innerHTML;
}
function cb(req)
{
if (req.status == 200)
debug('Received response: '+req.responseText);
else
debug('error '+req.status+' '+req.responseText);
}
function clicked()
{
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
send_request(cb, $('text').value);
}
test
""")
def process(request):
if request.POST.get('text'):
t =
models.Test.objects.get_or_create(str=request.POST.get('text'))[0]
return HttpResponse('%s %s' % (request.POST.get('text'), t.id))
else:
return HttpResponse('Missing argument')
--- CUT HERE ---
When I play around with it, I sometimes get this debug output followed
by an AssertionError (notice the different IDs):
20:07:41 Received response: blah 21
20:07:41 Received response: blah 19
20:07:41 Received response: blah 20
20:07:41 Received response: blah 19
I know that in this specific case it can be enforced at database level
using the unique and unique_together properties, but how would you
handle e.g. this scenario?
1 def transfer_money(from_id, to_id, amount)
2from = models.BankAccount.objects.get(id=from_id)
3to = models.BankAccount.objects.get(id=to_id)
4from.balance -= amount
5to.balance += amount
6from.save()
7to.save()
What, if two Django processes transfer money from account A to account
B simultaneously?
tom
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Re: Database race conditions when using multiple processes
On Oct 16, 2006, at 8:46 PM, gabor wrote: > > What, if two Django processes transfer money from account A to account > > B simultaneously? > > these situations are (afaik) handled using transactionssimply start > a transaction at the beginning, and commit/rollback at the end... I already included "django.middleware.transaction.TransactionMiddleware" in my MIDDLEWARE_CLASSES (as described on http://www.djangoproject.com/documentation/transactions/#tying-transactions-to-http-requests) and it didn't help. tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Database race conditions when using multiple processes
gabor wrote: > Thomas Steinacher wrote: > > I already included > > "django.middleware.transaction.TransactionMiddleware" in my > > MIDDLEWARE_CLASSES (as described on > > http://www.djangoproject.com/documentation/transactions/#tying-transactions-to-http-requests) > > and it didn't help. > > > > you mean it did not help with the money-transfer code? > > gabor It did not help with the code I posted that was using get_or_create(), because I didn't use unique_together... I don't think it would help with the money-transfer code. The documentation says: "It works like this: When a request starts, Django starts a transaction. If the response is produced without problems, Django commits any pending transactions. If the view function produces an exception, Django rolls back any pending transactions." The problem is that the code doesn't produce an exception, so Django will not roll back anything... tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Database race conditions when using multiple processes
James Bennett wrote: > On 10/16/06, gabor <[EMAIL PROTECTED]> wrote: > > would you use something in the db? (from what i know about transactions > > (very little :), they "solve" the potential conflicts by simply > > reporting an error-condition to one of the "writers", so then he has to > > retry. but is there some simply (multiprocess-usable) way to lock (as in > > "wait until this guy is finished")? > > http://www.postgresql.org/docs/8.1/static/sql-lock.html This is PostgreSQL specific. There seems also to be something like that for MySQL: http://dev.mysql.com/doc/refman/5.0/en/lock-tables.html (it looks like you have to do an UNLOCK TABLES on MySQL, whereas PostgreSQL releases the lock at transaction end). Shouldn't there be a function built-in into Django that locks the table (if the database supports it)? IMHO functions like get_or_create() should try to lock and unlock the table automatically. I am not doing critical operations like transferring money, but I noticed that the race condition happens quite often when using get_or_create and a lot of AJAX requests, so I need to find a solution. I will try using PostgreSQL instead of SQLite, maybe this will reduce the probability of the race condition to happen ;-) tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers -~--~~~~--~~--~--~---
Re: Database race conditions when using multiple processes
Michael Radziej wrote:
> It seems you need to spend more thoughts on multi user issues
> within your application. It's easy to fall for that in the
> beginning, but you need to deal with concurrent access. You must
> be aware for all transactions what impact other transactions can
> have. There is no silver bullet ... (please repeat ;-)
>
> > I will try using PostgreSQL instead of SQLite, maybe this will reduce
> > the probability of the race condition to happen ;-)
>
> Hmm, this might be a first step, but it won't magically solve all
> multi user issues. This *is* hard stuff.
Simply switching to PostgreSQL didn't solve anything. The race
condition was still there.
I am not sure but I think that a lock would be okay for my application,
as I don't want to deal with exceptions. What do you think about the
following code?
def process(request):
if request.POST.get('text'):
cursor = connection.cursor()
# Acquire a lock
cursor.execute('LOCK TABLE testapp_test IN ACCESS EXCLUSIVE
MODE')
t =
models.Test.objects.get_or_create(str=request.POST.get('text'))[0]
# Release the lock
transaction.commit()
return HttpResponse('%s %s' % (request.POST.get('text'), t.id))
else:
return HttpResponse('Missing argument')
process = transaction.commit_manually(process)
tom
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers
-~--~~~~--~~--~--~---
Bug in documentation
Hello,
I'm posting it here because Akismet is blocking my ticket. Can't you
use captchas instead of a stupid spam detector?
Session.objects.get_object() should be Session.objects.get()
On
http://www.djangoproject.com/documentation/sessions/#using-sessions-out-of-views
it says:
{{{
>>> from django.contrib.sessions.models import Session
>>> s = Session.objects.get_object(pk='2b1189a188b44ad18c35e113ac6ceead')
}}}
However, it should be:
{{{
>>> from django.contrib.sessions.models import Session
>>> s = Session.objects.get(pk='2b1189a188b44ad18c35e113ac6ceead')
}}}
tom
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
newforms:
Hello,
In newforms, there should be an easy way to render the radio fields
separately in the template, because I'd like to insert other form
fields between the radio inputs, e.g.:
[o] Use existing one: []
[ ] Add a new one: []
At the moment (using widget=forms.RadioSelect) all radio items are
rendered in a list and I can't insert anything between them.
I'd like to be able to do something like this:
{{ form.radio_options.1 }}: {{ form.foo }}
{{ form.radio_options.2 }}: {{ form.bar }}
What do you think?
Thomas Steinacher
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups "Django
developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
newforms: rendering radio fields separately
Hello,
In newforms, there should be an easy way to render the radio fields
separately in the template, because I'd like to insert other form
fields between the radio inputs, e.g.:
[o] Use existing one: []
[ ] Add a new one: []
At the moment (using widget=forms.RadioSelect) all radio items are
rendered in a list and I can't insert anything between them.
I'd like to be able to do something like this:
{{ form.radio_options.1 }}: {{ form.foo }}
{{ form.radio_options.2 }}: {{ form.bar }}
Or maybe something like this:
{% for opt in form.radio_options %}{{ opt }}{% endfor %}
What do you think?
Thomas Steinacher
P.S.: I sent this mail earlier but it didn't appear in the mailing list
for some reason... And google doesn't let me post using my e-mail
client :-(
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups "Django
developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
newforms: Creating a Widget with multiple HTML form elements
Hello,
Sometimes it would useful to create a FormField with a Widget that
consists of multiple HTML form elements. Rendering such a widget is
very easy, however, I don't see a possibility to get the data in the
clean() method of the Field. There should be an easy way to get the
data. Maybe by assigning multiple names for a field?
Consider the example below. I need this contact info in multiple forms
and I don't want to define all three fields in every form.
class ContactInfoWidget(forms.Widget):
def render(self, name, value, attrs=None):
print name
TYPES = (
(_('Main contact'), 'main_contact'),
(_('Billing contact'),'billing_contact'),
(_('Technical contact'), 'technical_contact'),
)
emails = ('[EMAIL PROTECTED]','[EMAIL PROTECTED]')
return '\n%s\n' % (
''.join([' %s: %s %s\n' %
(type, name, type_name, ''.join(['%s'
% email for email in emails]), _('Other...'))
for type, type_name in TYPES]))
class ContactInfoField(forms.Field):
widget = ContactInfoWidget
def __init__(self, required=True, widget=None, label=_('Contact
info')):
forms.Field.__init__(self, required, widget, label)
def clean(self, value):
print value
return value
Thomas Steinacher
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups "Django
developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: Is this group moderated, or is it a bug with google groups?
Yeah, google groups sucks. Using my gmail.com account it sometimes takes up to three days until a message appears. I also couldn't post to the list using a non-gmail account. It says that I'm not subscribed to the list, but I am. tom On Jan 26, 2007, at 7:49 AM, medhat wrote: > > So many times I send messages to the group, but my message does not > appear at all, or it might appear a day or two after I actually send > it, which of course makes it appear down on the list, and nobody > really > sees it. > > -- > Thanks, > Medhat > > > > --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Is this group moderated, or is it a bug with google groups?
This is funny. Now it seems to work. It didn't work for months. tom On Jan 26, 2007, at 6:30 PM, Thomas Steinacher wrote: > > Yeah, google groups sucks. Using my gmail.com account it sometimes > takes up to three days until a message appears. I also couldn't post > to the list using a non-gmail account. It says that I'm not > subscribed to the list, but I am. > > tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
newforms: Dynamic forms using JavaScript
Hello, I created a small library called "dynamicforms" (based on newforms) that allows you to edit multiple instances of an object simultaneously. It also provides a JavaScript link that allows you to add another instance of the object without having to reload the page. Take a look at it: http://eggdrop.ch/blog/2007/02/15/django-dynamicforms/ Maybe this could be improved and integrated into Django (see also ticket #3433)? Comments are appreciated. tom --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Proposal: Root view
Hello,
What is the current status of the ROOT_VIEW setting?
Are there any plans to implement this?
I am +1 on this, because it would allow me to use one Django instance
for multiple subdomains, without wasting system ressources. Also,
this is more flexible than e.g. a HostURLResolver (see http://
code.djangoproject.com/ticket/1192).
Thomas
On Nov 21, 2006, at 6:22 AM, Adrian Holovaty wrote:
>
> Currently, Django uses the ROOT_URLCONF to determine which module
> contains the URLconf from which to start processing the URL. Simon
> Willison and I were discussing this a while ago, and he had the idea
> of turning things around to make things more flexible.
>
> Instead of a ROOT_URLCONF setting, we could have a ROOT_VIEW setting,
> which would point to a single view function that acts as the site
> root. The URLconf, then, would *become* a view -- that is, patterns()
> would return an object whose __call__() method took an HttpRequest
> object and returned an HttpResponse. The logic of this view would
> simply implement the regular expression lookup stuff that the URLconf
> infrastructure currently uses.
>
> Why make this change? At first, it seems like it wouldn't be much
> different than the way Django currently works. But consider these
> advantages to a ROOT_VIEW setting:
>
> * All of a sudden URL traversal gets way more flexible. Your root view
> can delegate to Django's URLconf for a certain app, or some other URL
> traversal library (e.g., Routes) for another piece of the URL tree.
>
> For instance, you could set your root view to this:
>
> def my_root_view(request):
> if request.path.startswith('/foo/'):
> # Do something special
> return parse_urlconf(request, my_urlconf)
>
> (In this example, the framework would provide the parse_urlconf
> view function.)
>
> * The processing of requests gets conceptually simpler. The whole
> shebang is just a single view function, and the URLconf happens to be
> a "configurable" view function.
>
> * It would make it dead simple to do a web.py-ish "all in a single
> module" application, by providing a run_django() function that took a
> view and treated it as the root view:
>
> if __name__ == "__main__":
> run_django(root_view)
>
> This would also make testing simpler.
>
> * It would make view decorators more flexible. You could apply a view
> decorator to a selection of views, for instance. This might even make
> middleware unnecessary, but I haven't fully thought about that.
>
> * Oh, and it would be easy to make this fully backwards compatible.
>
> Thoughts, criticisms?
>
> Adrian
>
> --
> Adrian Holovaty
> holovaty.com | djangoproject.com
--~--~-~--~~~---~--~~
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-developers@googlegroups.com
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en
-~--~~~~--~~--~--~---
Re: newforms: compound or nested forms
+1 - Sub forms and form lists are great and IMHO very common. Please submit the patch. On Mar 9, 11:18 am, "Jeroen van Dongen" <[EMAIL PROTECTED]> wrote: > Apart from the SubForm "field" I've also created a FormList "field" > which takes a form definition, a min_count and a max_count which > allows for easy creation of a whole list of identical forms (think > edit_inline=models.STACKED). How would you render a FormList in a template? Would it be possible to render the FormList using JavaScript, to make it possible to add new items by clicking on a JavaScript link? I did something similar: http://eggdrop.ch/blog/2007/02/15/django-dynamicforms/ Maybe this could be combined... Thomas --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Templates: short comments {##} eats text
On Apr 26, 2007, at 4:40 PM, tonnzor wrote: > > This feature is demanded and it is easy to implement w/o side effects. > We should do it. +1 --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Templates: short comments {##} eats text
But then it should be at least mentioned in the documentation that this tag works only on a single line and will cause unexpected results when used on multiple lines (or better: raise an exception when using multiple lines). Thomas On Apr 26, 2007, at 9:46 PM, James Bennett wrote: > > On 4/26/07, tonnzor <[EMAIL PROTECTED]> wrote: >> This feature is demanded and it is easy to implement w/o side >> effects. >> We should do it. > > The ticket for this has been closed multiple times and Adrian has > asked that it not be reopened. Malcolm has asked that this thread be > allowed to die. That seems to indicate that a decision has been made > and that the proposed changes will not be implemented; regardless of > whether you agree with it or not, please respect that and allow the > list to move on to discussing other things. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
Re: Problems with FileFields/ImageFields
Hello, Maybe the snippet which I created today can help you. It displays a delete checkbox next to the : http://www.djangosnippets.org/snippets/469/ However, I'm not sure if it will work in admin. tom On 16.11.2007, at 18:12, Marc Garcia wrote: > > There are a couple of things about FileFields that I want to comment. > > First one is about how to delete content in a FileField. With a > example: > > I've a model: > > class Person(model.Model): >name = models.CharField(maxlength=32) >[...] >picture = models.ImageField(upload_to='pictures', blank=True) > > Imagine that I've created a person record with a picture of anoother > person, and I want to delete it (the picture). How can I do? I think > that in admin it should be a checkbox for FileFields to remove > content. > > Second problem that I have. With an example as well. > > In last application I want my users to upload its own data, so I > create a page with a form generated with form_for_model for previous > model. > > When displayed, for a user that already exists, and already has a > picture, an empty is displayed for picture > field. User can't see it's own picture with admin's link, and when > saved, old picture is dropped. I think that it should work like in > admin (with delete checkbox). > > Please, comment me what you think about and if necessary I'll develop > fixing patches. > > Thanks! > Marc --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-developers@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/django-developers?hl=en -~--~~~~--~~--~--~---
