Re: CSRF and Forms
On Tue, Sep 7, 2010 at 8:38 AM, Russell Keith-Magee
wrote:
> Firstly, I'm not wild about "secure=request.validated". This looks
> like a really simple way for people to say "secure=True" as a way of
> "fixing" CSRF support that they can't get to work. The choice of
> argument on the attribute isn't optional -- it *must* be
> request.validated. So really, it's the request that is the argument
> that needs to be passed in. The good news on this point is that a
> "request aware form" is something that has been floated in other
> discussions recently. I'll be sure to raise it at the DjangoCon
> sprints this week as a topic for discussion.
Agree. In the other CSRF thread I provided a shorter, slightly safer,
cookie-less implementation that works withut Referer headers and
limits each form's lifespan to 30 minutes. I have since implemented a
test form that works like this:
form = Form(request.POST or None, request.FILES or None, request.META)
The form is able to both issue a signed CSRF token (you can just write
{{ form.as_p }} or {{ form.csrf_token }}) and validate it (checks if
POST contains a token, checks the signature and fails form validation
if needed). It could cooperate with the middleware to redirect to an
error if no token was present at all.
Having it this way is useful as it's possible for someone to keep a
form open longer than said 30 minutes and it's more practical to
simply ask for confirmation than to drop all the entered data and
redirect to an error page.
> Secondly, IMHO there is a lot of value in the fact that Django forces
> the raising of a 403, rather than a 200 with an error message on the
> page. This isn't a form input error. It's a catastrophic problem that
> should only be observed when the user is actually under attack, or if
> cookies aren't available. Displaying a CSRF failure in the same way as
> an error for not putting 12 digits in your credit card number strikes
> me as the wrong way to visualize the error case. It implies that the
> problem can be fixed by user interaction when it can't -- at least,
> not by fixing form inputs. In order to give instructions on why
> cookies are needed and how to enable them, we need a lot more real
> estate... like a CSRF view.
See above, I see the value of the error in case of a CSRF attack. In
such cases there will be no valid token present in the payload. My
implementation can however differentiate between a missing token, an
invalid one and one that is simply too old to accept it.
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Tue, Sep 7, 2010 at 8:29 AM, Thomas Guettler wrote: > If you have a daemon that lives forever, I would do it like this (untested > if this does not leave an idle transaction open): > The place where the daemon hangs around if nothing can be done lives must > not use the ORM. If there is something to be done, the daemon calls methods > that use the commit_on_success decorators. Unfortunately you don't always get to choose what to call with what decorators. For example we have a DBus daemon that acts as an RPC server. Creating a separate function for each select is not really practical :) -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF and Forms
On Tue, Sep 7, 2010 at 4:08 AM, Patryk Zawadzki wrote:
> Hi,
>
> Since CSRF is already being reafactored up-side down in the trunk, I
> thought it might be a good idea to propose a slight modification.
Erm... it is? That's news to me.
Django 1.2 introduced a bunch of very big changes, but those changes
are signed, sealed and delivered. I'm not aware of any major changes
that have already been made, or are planned.
> Currently CSRF either falls through to the resolved view function or
> calls settings.CSRF_FAILURE_VIEW. More than once I've found myself
> wanting something in between. In such cases I'd like the logic flow to
> be able to reach the view, just telling me that CSRF did not validate.
>
> Let's say we add a new attribute to the request, call it "validated"
> and make it default to True for GET and False for everything else. Now
> split the CSRF middleware into two separate pieces of code. One
> middleware that does the validation and sets request.validated to True
> on success. One middleware that checks for (request.method == 'POST'
> and not request.validated) and in such cases returns
> settings.CSRF_FAILURE_VIEW.
>
> "How is that useful?" I hear you ask.
>
> class SecureForm(forms.Form):
> def __init__(self, *args, **kwargs):
> self.secure = kwargs.pop('secure', False)
> return super(SecureForm, self).__init__(*args, **kwargs)
> def _clean_form(self, *args, **kwargs):
> if not self.secure:
> self._errors[NON_FIELD_ERRORS] = self.error_class([
> 'We could not confirm that the request originated from
> your machine. Please resubmit to continue.'
> ])
> else:
> super(SecureForm, self)._clean_form(self, *args, **kwargs)
>
> def MyForm(SecureForm):
> foo = forms.CharField()
>
> def my_view(request):
> myform = MyForm(request.POST or None, request.FILES or None,
> secure=request.validated)
> if myform.is_valid():
> # ...
> pass
> return direct_to_template(request, 'my.html', {'form': myform})
Hrm. I see what you're doing here.
Firstly, I'm not wild about "secure=request.validated". This looks
like a really simple way for people to say "secure=True" as a way of
"fixing" CSRF support that they can't get to work. The choice of
argument on the attribute isn't optional -- it *must* be
request.validated. So really, it's the request that is the argument
that needs to be passed in. The good news on this point is that a
"request aware form" is something that has been floated in other
discussions recently. I'll be sure to raise it at the DjangoCon
sprints this week as a topic for discussion.
Secondly, IMHO there is a lot of value in the fact that Django forces
the raising of a 403, rather than a 200 with an error message on the
page. This isn't a form input error. It's a catastrophic problem that
should only be observed when the user is actually under attack, or if
cookies aren't available. Displaying a CSRF failure in the same way as
an error for not putting 12 digits in your credit card number strikes
me as the wrong way to visualize the error case. It implies that the
problem can be fixed by user interaction when it can't -- at least,
not by fixing form inputs. In order to give instructions on why
cookies are needed and how to enable them, we need a lot more real
estate... like a CSRF view.
Yours,
Russ Magee %-)
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
Patryk Zawadzki wrote: > On Mon, Sep 6, 2010 at 4:47 PM, Robert Gravsjö wrote: >> Patryk Zawadzki skrev 2010-09-06 15.29: >>> The isolating transaction keeps going on until you either (1) commit, >>> (2) rollback or (3) disconnect. Django only commits/rollbacks the >>> transactions it explicitly starts, it does not care about the >>> implicitly started isolating transaction. That's what results in >>> " in transaction" and I can reproduce it with a two-line view >>> that does a simple SELECT with no transaction middleware involved. >> Can you please show me the code you're running to reproduce this? > > Right, I misremembered the original problem. I've now found the > testing environment. > > The problem is not with regular views but with Celery tasks, > long-running management commands such as daemons and any other place > where you access the ORM from outside of the usual > request→dispatcher→view→response flow I use the TransactionMiddleware for requests, and scripts which are started from the shell use the commit_on_success decorator. Utility methods that are used in both ways (shell and request) don't use transaction handling methods. Long running tasks are an exception, here I use something like this: for i, .. in enumerate(...): if i%1000==0: commit() If you have a daemon that lives forever, I would do it like this (untested if this does not leave an idle transaction open): The place where the daemon hangs around if nothing can be done lives must not use the ORM. If there is something to be done, the daemon calls methods that use the commit_on_success decorators. Thomas -- Thomas Guettler, http://www.thomas-guettler.de/ E-Mail: guettli (*) thomas-guettler + de -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: python 3.x
On Sun, Sep 5, 2010 at 5:05 PM, VernonCole wrote: > "Once we're at a Django 2.6 minimum supported version, using 2to3 to > maintain > parallel implementations becomes a lot easier." > > As much as I admire Russ, and I do, I don't think that the above > statement is correct. My apologies -- in the haste of getting a response out, I was a little lax in my choice of words. What you've described - i.e., * A single maintained 2.X source tree * An auto-generated 3.X source tree, * When 2.X support is dropped, the migration script is run one last time to migrate the source tree to be 3.X is pretty close to what I had in my head as the likely path. In practice, I'm sure there will be some complications, but we won't really know what they are until we get serious about 3.X support. > For a short time on the pywin32 team we tried to "maintain parallel > implementations" and found that it was a mistake we had to undo. The > correct approach is to maintain a SINGLE implementation -- in Python 2 > format -- and use 2to3 as a tool when the code happens to be running > on Python 3+. 2to3 should be run by distutils when it detects that > setup.py is being run by Python3. It should NOT be run manually by a > human. > > Then, some years in the future when the last Python 2.7 engine fades > away, you will run 2to3 once for the last time, and THEN maintain in > Python 3 format. You do NOT write your code with print() functions, > etc.. Simply roll any needed refactoring into the trunk at the > earliest opportunity, and make sure you don't break them during > maintenance. > > That's my advice from my experience. The code I am supporting runs on > any version of Python from 2.3 thru 3.1, including IronPython. It's good to know that there are people in the community that have done this in anger on other projects. If anyone can provide patches for refactors that are necessary in order to simplify the 3.X migration process, I'm happy to apply those patches -- and as I indicated, I already have applied a couple of patches for exactly this reason; for example changeset 13509 was to change the way we use sorting functions to avoid a keyword argument that has been deprecated for 3.X. Yours, Russ Magee %-) -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, 2010-09-06 at 21:11 +0200, Patryk Zawadzki wrote: > See above, I'm concerned about the case when the attacker is able to > perform a replay attack. If you validate CSRF against the client's IP > (and possibly against a TTL field¹), you effectively prevent that type > of attack. If the CSRF tokens were stored in the database (much like > sessions are), you'd gain additional security of all tokens expiring > after one use (or TTL reached). Replay attacks? 1) You didn't mention this before 2) This is out of scope for what the CSRF defences are about 3) SSL protects us: http://stason.org/TULARC/security/ssl-talk/4-1-Does-SSL-protect-users-from-replay-attack-by-eavesdropp.html If you are talking about the session cookie being stolen because it is leaked over HTTP, that again is out of scope for CSRF - it can only be addressed by using a secure cookie (SESSION_COOKIE_SECURE). > > A MitM cannot tamper with requests that are sent over SSL. The only > > exception to this is that if HTTP -> HTTPS POST requests are allowed, > > the MitM can indeed generate any request they like and get the user's > > browser to send it, with potentially damaging consequences. > > The MitM can also trick the user into POSTing same data over HTTP and > forge a HTTPS request of his own. That's why I propose using a > separate, secure CSRF cookie (using a different name) for HTTPS > requests. It doesn't matter what the name of the cookie is, the attacker can still set a cookie with that name. > > To stop > > this we, we simply refuse HTTP -> HTTPS POST requests. In the context of > > an HTTPS request from a browser, we can indeed trust the Referer header > > since no-one can tamper with it in transit, and the MitM will not be > > able to convincingly send the user's browser a page that looks like it > > comes from https://example.com. > > I believe you can use mere cookies to prevent this (see above) and > keep the site working for people who don't and can't send Referer > headers. I don't think so, for reasons given below. > > ¹ Consider the following example. It was based on the code I wrote to > handle the new Facebook API so it might be a bit of an overkill but it > shows it's possible. A simpler solution would be to return (TTL + ':' > + md5(token + user_ip + secret_key + TTL)). > You are proposing adding IP address as a means of identifying a user, but that is a bad idea on two counts: 1) The attacker can have the same (public) IP address as the victim if they are both behind the same firewall, and this will in fact be one of the most practical ways to launch a MitM + CSRF attack on HTTPS. 2) There is no guarantee that a user has a single IP addresses for the duration of his session. My mobile phone's IP address can change quite quickly. Once you've removed the IP address from the equation, the only barrier you are adding is a TTL. However, that is a very small barrier - any view that returns a POST form can easily be parsed to retrieve a valid CSRF token - we hand out valid CSRF tokens very easily, whether over HTTP or HTTPS. The attacker can use a token retrieved in that way. There is nothing to connect the CSRF token to the user, and this is *by design*, because: 1) In Django we have no way to identify a user. We don't want to rely on the session framework, because a site might not be using it. 2) Even if a site is using the session framework, we've had issues in the past with stale CSRF tokens caused by session cycling. Almost everything I've said to this point is explained in the document I linked before. Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
I ran into this issue with daemons/scheduled jobs and "idle in transaction" in Django quite a while ago. I have a nice writeup of the problem and solution on this StackOverflow question if anybody's interested: http://stackoverflow.com/questions/1303654/threaded-django-task-doesnt-automatically-handle-transactions-or-db-connections Ultimately it's a matter of being aware of what Django will, and will not, do for you. The lack of a request/response cycle from the daemon means that the usual connection-closing mechanism [signals.request_finished.connect(close_connection)] never gets triggered. Being a *web framework* it seems reasonable to me that it's built around requests/responses... - Gabriel On Sep 6, 10:09 am, Jordan wrote: > On Mon, 2010-09-06 at 16:47 +0200, Robert Gravsjö wrote: > > Can you please show me the code you're running to reproduce this? > > > Regards, > > roppert > > I've experienced what Patryk is describing here. It seems that the > Django ORM, when not explicitly in a transaction, doesn't commit or > rollback after each query. > > You can reproduce this simply in the dbshell. Go into your favorite > project and try this (replacing `myproject`, `myapp`, and `MyModel` > appropriately): > > from myproject.myapp import models > models.MyModel.objects.all()[:1] > > Now, in a superuser database connection, check the results of "select * > from pg_stat_activity." You should see the database connection from the > dbshell stuck in the " in transaction" state. > > Regards, > Jordan > > signature.asc > < 1KViewDownload -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 10:39 PM, Patryk Zawadzki wrote: > ... First stab at a patch attached. Did not try to run it yet so it might contain syntax errors. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en. django-signed-csrf.patch Description: Binary data
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 9:11 PM, Patryk Zawadzki wrote:
> Consider the following example. It was based on the code I wrote to
> handle the new Facebook API so it might be a bit of an overkill but it
> shows it's possible. A simpler solution would be to return (TTL + ':'
> + md5(token + user_ip + secret_key + TTL)).
>
> def encode_cookie(token):
> values = {
> 'TTL': time.time() + 60*60,
> 'token': token,
> }
> payload = simplejson.dumps(values)
> digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
> encoded_payload = base64.b64encode(payload)
> encoded_digest = base64.b64encode(digest)
> return '%s.%s' % (encoded_digest, encoded_payload)
>
> def decode_cookie(value):
> encoded_sig, encoded_payload = map(str, value.split('.', 1))
> sig = base64.b64decode(encoded_sig)
> payload = base64.b64decode(encoded_payload)
> data = simplejson.loads(payload)
> digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
> if str(digest) != sig:
> return None
> if data.get('TTL', 0) < time.time():
> return None
> return data.get('token', None)
Another approach would be not to use a cookie at all. For each {%
csrf_token %} use a slightly modified variant of the above
encode_cookie function with:
values = {
'host: request.META['HTTP_HOST'],
'scheme': request.is_secure(),
'user_ip': request.META['REMOTE_ADDR'],
'user_agent': request.META['HTTP_USER_AGENT'],
'ttl': time.time() + 30*60,
}
Then when handling a POST request, decipher the token and compare each
META field with the ones from the request and validate ttl against
time.time(). I believe it's not less secure than the current
implementation and solves two problems:
1) each form served gets its own ttl, an attacker can't keep pinging
the server to keep the token alive
2) each token serves for a single use and will inevitably timeout in
30 minutes while still allowing you to open two forms in two browser
tabs and submit each of them separately
Attaching such an encoded string to the request should be harmless, it
won't be much longer than the cookie header, will not be sent with
each request (only sent when rendering a form, only received with form
submission).
If you like this approach, I suggest letting the form both generate
and validate the token if the middleware is not active. See my other
CSRF thread for the use case.
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
CSRF and Forms
Hi,
Since CSRF is already being reafactored up-side down in the trunk, I
thought it might be a good idea to propose a slight modification.
Currently CSRF either falls through to the resolved view function or
calls settings.CSRF_FAILURE_VIEW. More than once I've found myself
wanting something in between. In such cases I'd like the logic flow to
be able to reach the view, just telling me that CSRF did not validate.
Let's say we add a new attribute to the request, call it "validated"
and make it default to True for GET and False for everything else. Now
split the CSRF middleware into two separate pieces of code. One
middleware that does the validation and sets request.validated to True
on success. One middleware that checks for (request.method == 'POST'
and not request.validated) and in such cases returns
settings.CSRF_FAILURE_VIEW.
"How is that useful?" I hear you ask.
class SecureForm(forms.Form):
def __init__(self, *args, **kwargs):
self.secure = kwargs.pop('secure', False)
return super(SecureForm, self).__init__(*args, **kwargs)
def _clean_form(self, *args, **kwargs):
if not self.secure:
self._errors[NON_FIELD_ERRORS] = self.error_class([
'We could not confirm that the request originated from
your machine. Please resubmit to continue.'
])
else:
super(SecureForm, self)._clean_form(self, *args, **kwargs)
def MyForm(SecureForm):
foo = forms.CharField()
def my_view(request):
myform = MyForm(request.POST or None, request.FILES or None,
secure=request.validated)
if myform.is_valid():
# ...
pass
return direct_to_template(request, 'my.html', {'form': myform})
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: python 3.x
Thanks everybody! while indeed it's clear django will not official run on 3.0 any soon, it's clearer to me why & how. yes I'm aware of __future__ import, though it's not really magic (eg. support for bytes / unicode types is more of a compatibility thing, for argparse python 2.7 minimum is necessary, etc.) but I'd rather struggle with these things than without Django :-) I'll see if there are ways for me to help... cheers, Stefano On Sep 5, 11:05 am, VernonCole wrote: > "Once we're at a Django 2.6 minimum supported version, using 2to3 to > maintain > parallel implementations becomes a lot easier." > > As much as I admire Russ, and I do, I don't think that the above > statement is correct. > > For a short time on the pywin32 team we tried to "maintain parallel > implementations" and found that it was a mistake we had to undo. The > correct approach is to maintain a SINGLE implementation -- in Python 2 > format -- and use 2to3 as a tool when the code happens to be running > on Python 3+. 2to3 should be run by distutils when it detects that > setup.py is being run by Python3. It should NOT be run manually by a > human. > > Then, some years in the future when the last Python 2.7 engine fades > away, you will run 2to3 once for the last time, and THEN maintain in > Python 3 format. You do NOT write your code with print() functions, > etc.. Simply roll any needed refactoring into the trunk at the > earliest opportunity, and make sure you don't break them during > maintenance. > > That's my advice from my experience. The code I am supporting runs on > any version of Python from 2.3 thru 3.1, including IronPython. > -- > Vernon Cole -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 8:25 PM, Luke Plant wrote:
> Almost everything in this discussion and all your misconceptions are
> covered here:
>
> http://code.djangoproject.com/wiki/CsrfProtection
I've read it before joining the discussion.
> For HTTP (not HTTPS), MitM attacks are out of scope for our CSRF
> protection, because a MitM sees everything in the clear and will easily
> be able to defeat any defences we put up. Your suggested alterations do
> nothing to stop that.
I wasn't targetting HTTP at all. But I can't agree that there's
nothing we can do.
We can encode TTL into the CSRF cookie¹ to make sure being able to
eavesdrop at one point in time does not grant you permanent access to
the server.
> In the context of HTTPS, the Referer header check does indeed add a
> necessary and effective protection. The nature of a CSRF attack is that
> the attacker has to gets the *user's* browser to make the request, not
> the attacker's.
Plain CSRF is not interesting to me, requiring any non-trivial token
is enough to prevent blind CSRF attacks.
I am more interested in preventing an attack that consists of
eavesdropping a cookie sent over HTTP and using it to forge a POST
using a secure connection.
> The curl command line says nothing at all - I'm well
> aware that the attacker can do that from their own machine if they want
> to, but it won't do them any good! I could not understand how you
> thought that showing a curl command line is at all relevant.
See above, I'm concerned about the case when the attacker is able to
perform a replay attack. If you validate CSRF against the client's IP
(and possibly against a TTL field¹), you effectively prevent that type
of attack. If the CSRF tokens were stored in the database (much like
sessions are), you'd gain additional security of all tokens expiring
after one use (or TTL reached).
> A MitM cannot tamper with requests that are sent over SSL. The only
> exception to this is that if HTTP -> HTTPS POST requests are allowed,
> the MitM can indeed generate any request they like and get the user's
> browser to send it, with potentially damaging consequences.
The MitM can also trick the user into POSTing same data over HTTP and
forge a HTTPS request of his own. That's why I propose using a
separate, secure CSRF cookie (using a different name) for HTTPS
requests. Even if you trick the user into sending data over HTTP, you
will not gain access to the secure token. Any views that require HTTPS
will not be exploited no matter what the attacker sends to the
browser.
> To stop
> this we, we simply refuse HTTP -> HTTPS POST requests. In the context of
> an HTTPS request from a browser, we can indeed trust the Referer header
> since no-one can tamper with it in transit, and the MitM will not be
> able to convincingly send the user's browser a page that looks like it
> comes from https://example.com.
I believe you can use mere cookies to prevent this (see above) and
keep the site working for people who don't and can't send Referer
headers.
> Please distinguish between the HTTP/HTTPS context if you've still got
> questions.
As I said, I am only concerned about using HTTP as an attack vector to
access HTTPS.
¹ Consider the following example. It was based on the code I wrote to
handle the new Facebook API so it might be a bit of an overkill but it
shows it's possible. A simpler solution would be to return (TTL + ':'
+ md5(token + user_ip + secret_key + TTL)).
def encode_cookie(token):
values = {
'TTL': time.time() + 60*60,
'token': token,
}
payload = simplejson.dumps(values)
digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
encoded_payload = base64.b64encode(payload)
encoded_digest = base64.b64encode(digest)
return '%s.%s' % (encoded_digest, encoded_payload)
def decode_cookie(value):
encoded_sig, encoded_payload = map(str, value.split('.', 1))
sig = base64.b64decode(encoded_sig)
payload = base64.b64decode(encoded_payload)
data = simplejson.loads(payload)
digest = hmac.new(settings.SECRET_KEY, payload, hashlib.sha256).digest()
if str(digest) != sig:
return None
if data.get('TTL', 0) < time.time():
return None
return data.get('token', None)
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, 2010-09-06 at 18:46 +0200, Patryk Zawadzki wrote: > On Mon, Sep 6, 2010 at 6:34 PM, Luke Plant wrote: > > OK, that does it. I call 'troll'. If you're not trolling, my > > apologies, but I have run out of energy trying to explain this to you. > > I'm still waiting for you to explain anything. > You said you were afraid of a third-party injecting cookies over HTTP > that would be then sent over HTTPS. > > I replied by telling you that currently it does not have to do any > injecting as it can just forge both the cookie and the token when > doing a malicious POST (attaching any other cookies and post data > needed to DoEvil™) -- current CSRF implementation does not stop such > attacks at all. > > Moreover I proposed a more secure cookie header that is both immune to > stealing (as it's calculated and checked using your IP address) and > forging (as you no longer ask the client to send you the same string > twice, both post payload and headers being in plaintext). > > The curl example was there to illustrate the problem of the current > implementation: a malicious client or proxy is free to replace both > the cookie and post payload and CSRF will validate. If I can capture a > HTTP request (the MitM case) and read the headers, it's just a matter > of taking the cookies, replacing the CSRF token cookie with "foo" and > I am free to make any request on the site I want for as long as I > want. Requiring me to send a "Referer" header is a mild inconvenience > at most (just send "https://domain.com/";). Almost everything in this discussion and all your misconceptions are covered here: http://code.djangoproject.com/wiki/CsrfProtection For HTTP (not HTTPS), MitM attacks are out of scope for our CSRF protection, because a MitM sees everything in the clear and will easily be able to defeat any defences we put up. Your suggested alterations do nothing to stop that. In the context of HTTPS, the Referer header check does indeed add a necessary and effective protection. The nature of a CSRF attack is that the attacker has to gets the *user's* browser to make the request, not the attacker's. The curl command line says nothing at all - I'm well aware that the attacker can do that from their own machine if they want to, but it won't do them any good! I could not understand how you thought that showing a curl command line is at all relevant. A MitM cannot tamper with requests that are sent over SSL. The only exception to this is that if HTTP -> HTTPS POST requests are allowed, the MitM can indeed generate any request they like and get the user's browser to send it, with potentially damaging consequences. To stop this we, we simply refuse HTTP -> HTTPS POST requests. In the context of an HTTPS request from a browser, we can indeed trust the Referer header since no-one can tamper with it in transit, and the MitM will not be able to convincingly send the user's browser a page that looks like it comes from https://example.com. Please distinguish between the HTTP/HTTPS context if you've still got questions. Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Mon, Sep 6, 2010 at 4:47 PM, Robert Gravsjö wrote: > Patryk Zawadzki skrev 2010-09-06 15.29: >> The isolating transaction keeps going on until you either (1) commit, >> (2) rollback or (3) disconnect. Django only commits/rollbacks the >> transactions it explicitly starts, it does not care about the >> implicitly started isolating transaction. That's what results in >> " in transaction" and I can reproduce it with a two-line view >> that does a simple SELECT with no transaction middleware involved. > Can you please show me the code you're running to reproduce this? Right, I misremembered the original problem. I've now found the testing environment. The problem is not with regular views but with Celery tasks, long-running management commands such as daemons and any other place where you access the ORM from outside of the usual request→dispatcher→view→response flow. These cases all end up with an isolating transaction spanning their whole life span. In case of daemons it results in permanently blocking the database structure (for example causing "VACUUM FULL" to hang). It would be more useful if you could explicitly enter_isolation_block() and leave_isolation_block() as needed (currently there is no way to commit the isolating transaction other than doing a raw SQL query or accessing psycopg internals). Sorry about the confusion, it's not really related to views. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Mon, 2010-09-06 at 16:47 +0200, Robert Gravsjö wrote: > Can you please show me the code you're running to reproduce this? > > Regards, > roppert I've experienced what Patryk is describing here. It seems that the Django ORM, when not explicitly in a transaction, doesn't commit or rollback after each query. You can reproduce this simply in the dbshell. Go into your favorite project and try this (replacing `myproject`, `myapp`, and `MyModel` appropriately): from myproject.myapp import models models.MyModel.objects.all()[:1] Now, in a superuser database connection, check the results of "select * from pg_stat_activity." You should see the database connection from the dbshell stuck in the " in transaction" state. Regards, Jordan signature.asc Description: This is a digitally signed message part
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 6:34 PM, Luke Plant wrote: > OK, that does it. I call 'troll'. If you're not trolling, my > apologies, but I have run out of energy trying to explain this to you. I'm still waiting for you to explain anything. You said you were afraid of a third-party injecting cookies over HTTP that would be then sent over HTTPS. I replied by telling you that currently it does not have to do any injecting as it can just forge both the cookie and the token when doing a malicious POST (attaching any other cookies and post data needed to DoEvil™) -- current CSRF implementation does not stop such attacks at all. Moreover I proposed a more secure cookie header that is both immune to stealing (as it's calculated and checked using your IP address) and forging (as you no longer ask the client to send you the same string twice, both post payload and headers being in plaintext). The curl example was there to illustrate the problem of the current implementation: a malicious client or proxy is free to replace both the cookie and post payload and CSRF will validate. If I can capture a HTTP request (the MitM case) and read the headers, it's just a matter of taking the cookies, replacing the CSRF token cookie with "foo" and I am free to make any request on the site I want for as long as I want. Requiring me to send a "Referer" header is a mild inconvenience at most (just send "https://domain.com/";). Your only reply is calling me a troll. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, 2010-09-06 at 18:14 +0200, Patryk Zawadzki wrote:
> On Mon, Sep 6, 2010 at 5:56 PM, Patryk Zawadzki wrote:
> > In fact current implementation is weaker than the old one. You are now
> > depending on the client delivering the challenge and the response,
> > both being sent over HTTP. An attacker no longer has to steal a valid
> > session cookie, it's enough to pick any string ("foo" will suffice)
> > and send it both in cookie headers and in POST payload.
>
> Mandatory demo attack:
>
> $ curl --cookie \
> csrftoken=foo \
> --form csrfmiddlewaretoken=foo \
> --referer https://example.com/innocent
> https://example.com/do/something/evil
>
> :)
OK, that does it. I call 'troll'. If you're not trolling, my
apologies, but I have run out of energy trying to explain this to you.
Luke
--
A mosquito cried out in pain:
"A chemist has poisoned my brain!"
The cause of his sorrow
was para-dichloro-
diphenyltrichloroethane
Luke Plant || http://lukeplant.me.uk/
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 5:56 PM, Patryk Zawadzki wrote:
> In fact current implementation is weaker than the old one. You are now
> depending on the client delivering the challenge and the response,
> both being sent over HTTP. An attacker no longer has to steal a valid
> session cookie, it's enough to pick any string ("foo" will suffice)
> and send it both in cookie headers and in POST payload.
Mandatory demo attack:
$ curl --cookie \
csrftoken=foo \
--form csrfmiddlewaretoken=foo \
--referer https://example.com/innocent
https://example.com/do/something/evil
:)
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 5:54 PM, Luke Plant wrote: > On Mon, 2010-09-06 at 16:42 +0200, Patryk Zawadzki wrote: >> If you use a separate *secure* cookie for CSRF over SSL then it's not >> possible to submit data from an unsafe page to a safe processor. >> Secure cookies are only sent to SSL targets so it's not possible to >> intercept the cookie by any means. > No - the 'secure' flag for cookies only stops the cookie being sent over > a non-secure connection. It *cannot* be used to stop them being *set* > over non-secure connections, which is what we need. And cookies without > the secure flag will still be sent over a secure connection. If you use separate cookie names for SSL and non-SSL requests, the SSL one will never be set over unsecure connections by Django itself. If you try to prevent the cookies from being set by a malicious proxy, the current security scheme is so weak, you cannot guard against it anyway. The attacked does not need to set a cookie or even steal it, it's enough for him to make a request forging both POST and the Cookie header - see my other reply in the thread. > And, once again, this is not about the cookie being intercepted, it is > about it being *set* over an HTTP connection and then being *sent* over > HTTPS. That part is taken care of above. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 4:42 PM, Patryk Zawadzki wrote:
> On Mon, Sep 6, 2010 at 3:55 PM, Luke Plant wrote:
>> On Sun, 2010-09-05 at 19:49 +0200, Patryk Zawadzki wrote:
>>> As for the vulnerability -- it's only there if you implement it
>>> yourself. If you send the initial login form over SSL (we do it this
>>> way for various reasons), the cookies are never prone to be
>>> intercepted.
>> No, the vulnerability we are talking about is there even if *you* don't
>> send the forms over HTTP, because the MitM can insert the forms
>> themselves (and send a matching CSRF cookie) into any page served over
>> HTTP, and have those forms target HTTPS connections, and submit them
>> automatically by javascript. It's not about cookies being intercepted,
>> it's about them being set by a MitM over HTTP.
> Of course if this is a true MitM attack, the attacker could just take
> your cookie and forge a request of his own, including a perfectly fine
> REFERER header.
Actually after reading the code for the current implementation¹ it
seems it does nothing to prevent MitM attacks. It only prevents basic
CSRF attacks where the attacker is not able to eavesdrop the
request/response cycle. The site still needs to rely on mechanisms
such as django.contrib.session to prevent MitM attacks.
In fact current implementation is weaker than the old one. You are now
depending on the client delivering the challenge and the response,
both being sent over HTTP. An attacker no longer has to steal a valid
session cookie, it's enough to pick any string ("foo" will suffice)
and send it both in cookie headers and in POST payload.
I'd suggest setting the cookie to something like:
md5(META["CSRF_TOKEN"] + META["REMOTE_ADDR"] + settings.SECRET_KEY)
¹ http://code.djangoproject.com/browser/django/trunk/django/middleware/csrf.py
--
Patryk Zawadzki
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To post to this group, send email to django-develop...@googlegroups.com.
To unsubscribe from this group, send email to
django-developers+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, 2010-09-06 at 16:42 +0200, Patryk Zawadzki wrote: > If you use a separate *secure* cookie for CSRF over SSL then it's not > possible to submit data from an unsafe page to a safe processor. > Secure cookies are only sent to SSL targets so it's not possible to > intercept the cookie by any means. No - the 'secure' flag for cookies only stops the cookie being sent over a non-secure connection. It *cannot* be used to stop them being *set* over non-secure connections, which is what we need. And cookies without the secure flag will still be sent over a secure connection. And, once again, this is not about the cookie being intercepted, it is about it being *set* over an HTTP connection and then being *sent* over HTTPS. Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
Patryk Zawadzki skrev 2010-09-06 15.29: On Mon, Sep 6, 2010 at 2:00 PM, Robert Gravsjö wrote: I'm not sure what you think you are doing but if you end up with " in transaction" that means you have not commited your transactions. See below. For instance, open two connections with psql and run BEGIN in one and then take a look at pg_stat_activity you will have that connection marked as idle in transaction. I believe you are confused about isolation levels. They control the visibility of transactions. Maybe I am indeed confused but quoting the psycopg documentation¹: "psycopg2.extensions.ISOLATION_LEVEL_READ_COMMITTED: This is the default value. A new transaction is started at the first execute() command on a cursor and at each new execute() after a commit() or a rollback(). The transaction runs in the PostgreSQL READ COMMITTED isolation level." The isolating transaction keeps going on until you either (1) commit, (2) rollback or (3) disconnect. Django only commits/rollbacks the transactions it explicitly starts, it does not care about the implicitly started isolating transaction. That's what results in " in transaction" and I can reproduce it with a two-line view that does a simple SELECT with no transaction middleware involved. Can you please show me the code you're running to reproduce this? Regards, roppert The problem only exists when Django sets isolation level to 1, if you use the deprecated "autocommit" setting, you will not be affected. ¹ http://initd.org/psycopg/docs/extensions.html#isolation-level-constants -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Mon, Sep 6, 2010 at 3:55 PM, Luke Plant wrote: > On Sun, 2010-09-05 at 19:49 +0200, Patryk Zawadzki wrote: >> As for the vulnerability -- it's only there if you implement it >> yourself. If you send the initial login form over SSL (we do it this >> way for various reasons), the cookies are never prone to be >> intercepted. > No, the vulnerability we are talking about is there even if *you* don't > send the forms over HTTP, because the MitM can insert the forms > themselves (and send a matching CSRF cookie) into any page served over > HTTP, and have those forms target HTTPS connections, and submit them > automatically by javascript. It's not about cookies being intercepted, > it's about them being set by a MitM over HTTP. If you use a separate *secure* cookie for CSRF over SSL then it's not possible to submit data from an unsafe page to a safe processor. Secure cookies are only sent to SSL targets so it's not possible to intercept the cookie by any means. I suggest you enforce that by default (if CSRF_ALTERNATIVE is False). Now if CSRF_ALTERNATIVE is True, feel free to use the same cookie for HTTP and HTTPS and rely on the REFERER header. This will of course break for a small minority of internet users, probably tech-savvy ones who are respected enough to convince other people not to use my application. Of course if this is a true MitM attack, the attacker could just take your cookie and forge a request of his own, including a perfectly fine REFERER header. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: CSRF Middleware/SSL/Firefox 3.6.8 bug
On Sun, 2010-09-05 at 19:49 +0200, Patryk Zawadzki wrote: > As for the vulnerability -- it's only there if you implement it > yourself. If you send the initial login form over SSL (we do it this > way for various reasons), the cookies are never prone to be > intercepted. No, the vulnerability we are talking about is there even if *you* don't send the forms over HTTP, because the MitM can insert the forms themselves (and send a matching CSRF cookie) into any page served over HTTP, and have those forms target HTTPS connections, and submit them automatically by javascript. It's not about cookies being intercepted, it's about them being set by a MitM over HTTP. There may be mitigating factors that limit the damage (like if you have sent your session cookies with 'secure only'), but our CSRF protection aims to be generic and not depend on those details, and also to provide protection against login CSRF (even if you are not using Django's session framework). Luke -- A mosquito cried out in pain: "A chemist has poisoned my brain!" The cause of his sorrow was para-dichloro- diphenyltrichloroethane Luke Plant || http://lukeplant.me.uk/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Mon, Sep 6, 2010 at 2:00 PM, Robert Gravsjö wrote: > I'm not sure what you think you are doing but if you end up with " > in transaction" that means you have not commited your transactions. See below. > For instance, open two connections with psql and run BEGIN in one and > then take a look at pg_stat_activity you will have that connection > marked as idle in transaction. > > I believe you are confused about isolation levels. They control the > visibility of transactions. Maybe I am indeed confused but quoting the psycopg documentation¹: "psycopg2.extensions.ISOLATION_LEVEL_READ_COMMITTED: This is the default value. A new transaction is started at the first execute() command on a cursor and at each new execute() after a commit() or a rollback(). The transaction runs in the PostgreSQL READ COMMITTED isolation level." The isolating transaction keeps going on until you either (1) commit, (2) rollback or (3) disconnect. Django only commits/rollbacks the transactions it explicitly starts, it does not care about the implicitly started isolating transaction. That's what results in " in transaction" and I can reproduce it with a two-line view that does a simple SELECT with no transaction middleware involved. The problem only exists when Django sets isolation level to 1, if you use the deprecated "autocommit" setting, you will not be affected. ¹ http://initd.org/psycopg/docs/extensions.html#isolation-level-constants -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
Patryk Zawadzki skrev 2010-09-06 12.20: On Mon, Sep 6, 2010 at 11:20 AM, Kirit Sælensminde (kayess) wrote: --- snip --- You can check this by creating a fresh project using psycopg2 and creating a model. Then write a view that queries the database and invoke it. Now, leaving the server running, open up your database shell. The pg_stat_activity table will report " in transaction" for hours. I'm not sure what you think you are doing but if you end up with " in transaction" that means you have not commited your transactions. For instance, open two connections with psql and run BEGIN in one and then take a look at pg_stat_activity you will have that connection marked as idle in transaction. I believe you are confused about isolation levels. They control the visibility of transactions. Regards, roppert -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Mon, Sep 6, 2010 at 1:35 PM, burc...@gmail.com wrote: > Hi Patryk, > > This was done to make Django faster, so it doesn't create connection > to database every time new SQL is executed. What was done? Commiting the isolating transaction of a connection does not terminate it. > Do you mean you wanted to set up timeouts for idle database connections? > I guess, nobody asked this before. No, I think I stated quite clearly what my proposal was. http://initd.org/psycopg/docs/extensions.html#isolation-level-constants -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
Hi Patryk, This was done to make Django faster, so it doesn't create connection to database every time new SQL is executed. Do you mean you wanted to set up timeouts for idle database connections? I guess, nobody asked this before. On Mon, Sep 6, 2010 at 5:20 PM, Patryk Zawadzki wrote: > On Mon, Sep 6, 2010 at 11:20 AM, Kirit Sælensminde (kayess) > wrote: >> Strange. We use Postgres and don't see any problem with this. We do >> encounter complications occasionally with the lack of composition of >> Django transaction handling, but other than that find the transaction >> handling adequate. >> >> Are you actually using transactions in your code? You need to be. The >> transaction middleware does a good enough job for most things, but for >> external processing you will need to carefully design your transaction >> handling. With the transaction middleware we see fully isolated >> updates until the transaction is committed using psycopg2. > > I'm not sure you understand the problem at all. The problem is not > lack of the isolation. The problem is permanent isolation. The > isolating transaction is never terminated, thus remaining alive for > indefinite amounts of time. > > You can check this by creating a fresh project using psycopg2 and > creating a model. Then write a view that queries the database and > invoke it. Now, leaving the server running, open up your database > shell. The pg_stat_activity table will report " in transaction" > for hours. > > BTW: This has nothing to do with Django transactions or TransactionMiddleware. > > -- > Patryk Zawadzki > > -- > You received this message because you are subscribed to the Google Groups > "Django developers" group. > To post to this group, send email to django-develop...@googlegroups.com. > To unsubscribe from this group, send email to > django-developers+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/django-developers?hl=en. > > -- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov, MSN: bu...@live.com -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
On Mon, Sep 6, 2010 at 11:20 AM, Kirit Sælensminde (kayess) wrote: > Strange. We use Postgres and don't see any problem with this. We do > encounter complications occasionally with the lack of composition of > Django transaction handling, but other than that find the transaction > handling adequate. > > Are you actually using transactions in your code? You need to be. The > transaction middleware does a good enough job for most things, but for > external processing you will need to carefully design your transaction > handling. With the transaction middleware we see fully isolated > updates until the transaction is committed using psycopg2. I'm not sure you understand the problem at all. The problem is not lack of the isolation. The problem is permanent isolation. The isolating transaction is never terminated, thus remaining alive for indefinite amounts of time. You can check this by creating a fresh project using psycopg2 and creating a model. Then write a view that queries the database and invoke it. Now, leaving the server running, open up your database shell. The pg_stat_activity table will report " in transaction" for hours. BTW: This has nothing to do with Django transactions or TransactionMiddleware. -- Patryk Zawadzki -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
Re: #13870: Correctly handling database isolation (in PostgreSQL)
Strange. We use Postgres and don't see any problem with this. We do encounter complications occasionally with the lack of composition of Django transaction handling, but other than that find the transaction handling adequate. Are you actually using transactions in your code? You need to be. The transaction middleware does a good enough job for most things, but for external processing you will need to carefully design your transaction handling. With the transaction middleware we see fully isolated updates until the transaction is committed using psycopg2. Kirit On Sep 5, 6:52 pm, Patryk Zawadzki wrote: > The problem is that this is not true for Django. The backend > initializes the meta-transaction at connection time and never bothers > to terminate it. As the transaction is merely a ghost-read-preventing > one, it does not stop data from being saved in the database, but it > does result in a parmanently open transaction. This both wastes > resources on the server and prevents any other client from touching > the database structure (any ALTER or VACUUM command will hang waiting > for the remaining transactions to finish). > > The ticket contains a naïve workaround but as described there, I feel > a proper solution would look like this: > > 1. Introduce IsolationMiddleware that does something like this (pseudocode): [snip] -- You received this message because you are subscribed to the Google Groups "Django developers" group. To post to this group, send email to django-develop...@googlegroups.com. To unsubscribe from this group, send email to django-developers+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
