Re: Support Negative Indexing on QuerySets
Hi Wim,
On Wednesday, July 31, 2013 12:04:42 AM UTC+2, Wim Lewis wrote:
>
> On 30 Jul 2013, at 2:06 PM, Florian Apolloner wrote:
> > How do you think such support would look like? For negative indices
> you'd have to know the size of the resultset to be able to do "limit
> somthing offset length-your_negative_index" -- this doesn't seem to make
> any sense for an ORM. You can always do list(qs)]:-1] though…
>
> It seems like the first comment in the ticket answers that question.
> Django would reverse the sense of the query's ordering clause and use a
> simple LIMIT.
>
In my opinion it doesn't; eg imagine the following as query:
MyModel.objects.order_by('whatever')[0:-50]; this isn't translateable into
MyModel.objects.order_by('-whatever')[50:] (the issue here is that the end
is now again undefined) since at least SQLite doesn't support an OFFSET
clause without a LIMIT. Also I think it's more natural to rewrite the
ordering of the query yourself to express that instead of using negative
ranges.
If there isn't an ordering clause in the query, then I agree it makes no
> sense to do any indexing other than [0:N].
>
In that case it's even debatable if limiting makes any sense at all ;)
Cheers,
Florian
--
You received this message because you are subscribed to the Google Groups
"Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to django-developers+unsubscr...@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Wednesday, July 31, 2013 1:03:31 AM UTC+2, Andre Terra wrote: > > On Tue, Jul 30, 2013 at 6:55 PM, Florian Apolloner > > > wrote: > >> Right, it depends on your usecase; I was just trying to point out other >> alternatives aside from the ones mentioned on the ticket. > > > I'm sorry if I seemed arrogant in my post. I most definitely did not > intend it, as I was absolutely sure that you were aware of the implications > of wrapping list() around a queryset. No worries, you weren't sry if my wording suggested that. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Tue, Jul 30, 2013 at 6:55 PM, Florian Apolloner wrote: > Right, it depends on your usecase; I was just trying to point out other > alternatives aside from the ones mentioned on the ticket. I'm sorry if I seemed arrogant in my post. I most definitely did not intend it, as I was absolutely sure that you were aware of the implications of wrapping list() around a queryset. My motivation was solely to highlight that caveat for potential lurkers and future readers visiting this thread that quite probably will not be as experienced as you are. Best wishes, AT -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On 30 Jul 2013, at 2:06 PM, Florian Apolloner wrote: > How do you think such support would look like? For negative indices you'd > have to know the size of the resultset to be able to do "limit somthing > offset length-your_negative_index" -- this doesn't seem to make any sense for > an ORM. You can always do list(qs)]:-1] though… It seems like the first comment in the ticket answers that question. Django would reverse the sense of the query's ordering clause and use a simple LIMIT. If there isn't an ordering clause in the query, then I agree it makes no sense to do any indexing other than [0:N]. ubernostrum's comment in the ticket makes it sound as if there is some explicit reasoning behind this, though, that's at least three years old. We can speculate, but does anyone know what that reasoning actually was? -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Tuesday, July 30, 2013 11:34:18 PM UTC+2, Andre Terra wrote: > > On Tue, Jul 30, 2013 at 6:06 PM, Florian Apolloner > > > wrote: > >> You can always do list(qs)]:-1] though… > > > Although you really shouldn't [1]. Right, it depends on your usecase; I was just trying to point out other alternatives aside from the ones mentioned on the ticket. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
On Tue, Jul 30, 2013 at 6:06 PM, Florian Apolloner wrote: > You can always do list(qs)]:-1] though… Although you really shouldn't [1]. As for the reasons for disallowing negative indexes, dcramer's comment in the ticket makes it clear: there is no way to infer what the last item in a query would be, except if you order it descendingly. For what is worth, production code should never rely on any kind of indexing that's not accompanied by an explicit order-by clause, as the default ordering is unrealiable -- at least in PostgreSQL[2], and I assume in other vendors as well[3]. In fact, one might even argue that it would make more sense to disallow any sort of indexing on querysets lacking an explicit .order_by(), in order to enforce best practices, although I would find that a little too strict. The specific one-off case of retrieving a single row for testing/developing/debugging would be best served by the use of .first() which orders by the primary_key by default[4], which therefore yields a different query from the one created by queryset indexing[5]. As it currently stands, the API mirrors the experience in writing a raw query insofar as it is *impossible* to retrieve the last row of a query without explicitly defining an order by clause. For this reason, I'm -1 on the proposed change. Cheers, AT [1] https://docs.djangoproject.com/en/dev/ref/models/querysets/ [2] http://www.postgresql.org/docs/9.2/static/queries-order.html [3] http://stackoverflow.com/a/5628674/447485 [4] https://docs.djangoproject.com/en/dev/ref/models/querysets/#first [5] http://dpaste.com/hold/1323698/ -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Support Negative Indexing on QuerySets
Hi Mark, How do you think such support would look like? For negative indices you'd have to know the size of the resultset to be able to do "limit somthing offset length-your_negative_index" -- this doesn't seem to make any sense for an ORM. You can always do list(qs)]:-1] though… Cheers, Florian -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Support Negative Indexing on QuerySets
In this support ticket (https://code.djangoproject.com/ticket/13089), the original closer said "This is a long-standing explicit design decision in the ORM, and so gets a wontfix...". What is the reasoning behind this? Why is negative indexing of querysets disallowed? Thanks! -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
Re: Proposal: Modifying the CSRF protection scheme
I agree with Jacob on both points. +1 from me, especially since neither of these changes should require changes in application code which is already using the interface correctly. -Paul On Tue, Jul 30, 2013 at 1:22 PM, Jacob Kaplan-Moss wrote: > Hey Shai - > > I have no objections to this change. I think it's got a slight whiff of > security theatre, in that it *looks* like it adds more protection than it > *actually* does. However, I, too, have spent a ton of time talking auditors > down from "OMG Django is vulnerable to CSRF!" and I'd like to do less of > that. I like that rotating SECRET_KEY invalidates CSRF tokens. > > Time-limiting is a nice feature too, actually. Again the perceived security > is higher than the actual added security, but the general principle of > giving people more control is a good one. I'm sure there's some audit > checklist out there that has "CSRF tokens must not be valid for longer than > X hours" or something on it, and helping our users tick those boxes isn't > such a bad thing. > > So yeah, lukewarm praise from me at best, but since there's at least a bit > of real improvement here I see no reason this shouldn't go in. +1 from me. > > Jacob > > > On Sat, Jul 27, 2013 at 6:12 PM, Shai Berger wrote: >> >> Hi everybody, >> >> TL;DR: A simple change can make Django's CSRF protection a little better; >> an >> additional, slightly less simple one, can also make it look better. >> >> Django's CSRF protection scheme is a bit unusual; unlike most such >> schemes, it >> does not rely on a value stored in the server that needs to be matched by >> a >> submitted token and is replaced with every submission, but rather on a >> constant value stored in a cookie. This generally works (for details of >> how >> and under what conditions exactly, see [1]), but has two minor problems: >> >> 1) It is unusual, and in particular diverges from what OWASP[2] >> recommends[3]; >> as a result, security analysts often think it is not secure. They have >> been >> proven wrong in all cases members of core are aware of, but proving it >> again >> and again is a nuisance, and there may be bad PR related to this. >> >> 2) It carries a "second-order" vulnerability: If your site has been >> compromised (XSS, Man-in-the-middle, or server compromise) then you become >> persistently vulnerable to CSRF. All of these vulnerabilities are way >> worse >> than CSRF and render all CSRF protection schemes worthless while they >> last; >> the point is *not* that they allow CSRF, but rather that they allow CSRF >> to be >> performed after the main hole has been plugged. This is because the >> attacker >> can use the main vulnerability to "steal", or even set, csrftoken cookie >> values, which they can then use later. After a successful attack of this >> magnitude, you need to reset the csrftoken cookies of all users, and this >> is >> neither obvious nor straightforward to do. >> >> Django's unique scheme does have two advantages over the more common >> solutions, which we would like to keep: >> >> 1) It is not tied to sessions, users, or site-stored per-user data, >> allowing >> CSRF protection to a wider range of users >> >> 2) It avoids the problem of having only one "current" token, which causes >> the >> submission of one form to invalidate forms open in other browser tabs. >> >> To improve on both problem issues, while keeping the advantages, I suggest >> the >> following modifications: >> >> a) Use a signed cookie for csrftoken -- using Django's existing signing >> facility[4], this means signing the cookie with the SECRET_KEY from the >> settings; so that an attacker cannot set arbitrary cookies, and changing >> the >> SECRET_KEY after a compromise immeiately invalidates csrftoken cookies. >> >> b) Optionally allowing time-limited CSRF tokens. Such tokens will be >> generated >> by adding a parameter of maximum age to the csrftoken tag, and by marking >> view >> methods (specifically with a decorator, or globally with a setting) as >> requiring timed tokens. When this is used, the posted token value will >> need to >> be different from the cookie value -- to keep advantage 2, the cookie will >> still be constant, and expiry time will only be present in the submitted >> token[5]. This method breaks the current way we do CSRF-protected AJAX, so >> it >> will likely stay optional (and opt-in). >> >> As you may guess, signing the cookie adds an actual iota of security. >> Adding >> expiry adds very little -- if an attacker has access to the cookie, they >> can >> usually just ask the site to generate valid tokens for them, so getting >> any >> real protection will require annoyingly short expiry times. But the fact >> that >> an attacker needs this extra step makes it a tiny bit harder for them and >> makes their actions a tiny bit more detectable; and having a constantly- >> changing CSRF token may make the whole thing look a little better to naive >> analysts. >> >> I had some help and guidance in drafting th
Re: Proposal: Modifying the CSRF protection scheme
Hey Shai - I have no objections to this change. I think it's got a slight whiff of security theatre, in that it *looks* like it adds more protection than it *actually* does. However, I, too, have spent a ton of time talking auditors down from "OMG Django is vulnerable to CSRF!" and I'd like to do less of that. I like that rotating SECRET_KEY invalidates CSRF tokens. Time-limiting is a nice feature too, actually. Again the perceived security is higher than the actual added security, but the general principle of giving people more control is a good one. I'm sure there's some audit checklist out there that has "CSRF tokens must not be valid for longer than X hours" or something on it, and helping our users tick those boxes isn't such a bad thing. So yeah, lukewarm praise from me at best, but since there's at least a bit of real improvement here I see no reason this shouldn't go in. +1 from me. Jacob On Sat, Jul 27, 2013 at 6:12 PM, Shai Berger wrote: > Hi everybody, > > TL;DR: A simple change can make Django's CSRF protection a little better; > an > additional, slightly less simple one, can also make it look better. > > Django's CSRF protection scheme is a bit unusual; unlike most such > schemes, it > does not rely on a value stored in the server that needs to be matched by a > submitted token and is replaced with every submission, but rather on a > constant value stored in a cookie. This generally works (for details of how > and under what conditions exactly, see [1]), but has two minor problems: > > 1) It is unusual, and in particular diverges from what OWASP[2] > recommends[3]; > as a result, security analysts often think it is not secure. They have been > proven wrong in all cases members of core are aware of, but proving it > again > and again is a nuisance, and there may be bad PR related to this. > > 2) It carries a "second-order" vulnerability: If your site has been > compromised (XSS, Man-in-the-middle, or server compromise) then you become > persistently vulnerable to CSRF. All of these vulnerabilities are way worse > than CSRF and render all CSRF protection schemes worthless while they last; > the point is *not* that they allow CSRF, but rather that they allow CSRF > to be > performed after the main hole has been plugged. This is because the > attacker > can use the main vulnerability to "steal", or even set, csrftoken cookie > values, which they can then use later. After a successful attack of this > magnitude, you need to reset the csrftoken cookies of all users, and this > is > neither obvious nor straightforward to do. > > Django's unique scheme does have two advantages over the more common > solutions, which we would like to keep: > > 1) It is not tied to sessions, users, or site-stored per-user data, > allowing > CSRF protection to a wider range of users > > 2) It avoids the problem of having only one "current" token, which causes > the > submission of one form to invalidate forms open in other browser tabs. > > To improve on both problem issues, while keeping the advantages, I suggest > the > following modifications: > > a) Use a signed cookie for csrftoken -- using Django's existing signing > facility[4], this means signing the cookie with the SECRET_KEY from the > settings; so that an attacker cannot set arbitrary cookies, and changing > the > SECRET_KEY after a compromise immeiately invalidates csrftoken cookies. > > b) Optionally allowing time-limited CSRF tokens. Such tokens will be > generated > by adding a parameter of maximum age to the csrftoken tag, and by marking > view > methods (specifically with a decorator, or globally with a setting) as > requiring timed tokens. When this is used, the posted token value will > need to > be different from the cookie value -- to keep advantage 2, the cookie will > still be constant, and expiry time will only be present in the submitted > token[5]. This method breaks the current way we do CSRF-protected AJAX, so > it > will likely stay optional (and opt-in). > > As you may guess, signing the cookie adds an actual iota of security. > Adding > expiry adds very little -- if an attacker has access to the cookie, they > can > usually just ask the site to generate valid tokens for them, so getting any > real protection will require annoyingly short expiry times. But the fact > that > an attacker needs this extra step makes it a tiny bit harder for them and > makes their actions a tiny bit more detectable; and having a constantly- > changing CSRF token may make the whole thing look a little better to naive > analysts. > > I had some help and guidance in drafting this proposal -- you can credit > Donald Stufft, mostly, for any egregious blunder I didn't make. I am still > responsible for the ones I did make. > > Your comments are welcome, > > Shai. > > > [1] https://docs.djangoproject.com/en/dev/ref/contrib/csrf/ -- in > particular, > "how it works" and "limitations" > > [2] https://www.owasp.org > > [3] > https://www.owasp.org/index.ph
Re: Avoid unbounded memory consumption when running `manage.py test`
Hi Matt, But I'm interested to know if others run into similar problems and how you > solve them. > We hit the same problem and used a similar tearDownClass solution http://chrisbailey.blogs.ilrt.org/2013/05/19/pythons-leaky-testcase-aka-hidden-gotchas-using-self/ Chris. -- You received this message because you are subscribed to the Google Groups "Django developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To post to this group, send email to django-developers@googlegroups.com. Visit this group at http://groups.google.com/group/django-developers. For more options, visit https://groups.google.com/groups/opt_out.
