Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Christophe Pettus
On Nov 23, 2014, at 1:53 PM, Rick van Hattem wrote: > Very true, that's a fair point. That's why I'm opting for a configurable > option. Patching this within Django has saved me in quite a few cases but it > can have drawbacks. As a DB guy, I have to say that if an application is sending a qu

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Rick van Hattem
On 23 Nov 2014 22:13, "Christophe Pettus" wrote: > > > On Nov 23, 2014, at 1:07 PM, Rick van Hattem wrote: > > > > Not really, cause psycopg already fetched everything. > > > > Not if Django limits it by default :) > > Unfortunately, that's not how it works. There are three things that take up m

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Christophe Pettus
On Nov 23, 2014, at 1:07 PM, Rick van Hattem wrote: > > Not really, cause psycopg already fetched everything. > > Not if Django limits it by default :) Unfortunately, that's not how it works. There are three things that take up memory as the result of a query result: 1. The Django objects.

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Rick van Hattem
Hi Florian, On 23 Nov 2014 16:22, "Florian Apolloner" wrote: > > Hi Rick, > > > On Sunday, November 23, 2014 1:11:13 PM UTC+1, Rick van Hattem wrote: >> >> If/when an unsliced queryset were to reach a certain limit (say, 10,000, but configurable) the system would raise an error. > > > Django can'

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Marco Paolini
2014-11-23 13:27 GMT+01:00 Shai Berger : > Hi Rick, > > On Sunday 23 November 2014 14:11:13 Rick van Hattem wrote: > > > > So please, can anyone give a good argument as to why any sane person > would > > have a problem with a huge default limit which will kill the performance > of > > your site an

Fellow Report: Nov 23, 2014

2014-11-23 Thread Berker Peksağ
Hi all, Here is this week's report. You can see a more readable version of this report at https://code.djangoproject.com/wiki/DjangoFellows/BerkerPeksag#Weekof2014-11-17 Pull requests - Authored ^ * [Committed] https://github.com/django/django/pull/3582 -- Fixed #12098

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Florian Apolloner
Hi Rick, On Sunday, November 23, 2014 1:11:13 PM UTC+1, Rick van Hattem wrote: > If/when an *unsliced* queryset were to reach a certain limit (say, > 10,000, but configurable) the system would raise an error. > Django can't know if that would be the case without issuing an extra query -- and e

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Shai Berger
Hi Rick, On Sunday 23 November 2014 14:11:13 Rick van Hattem wrote: > > So please, can anyone give a good argument as to why any sane person would > have a problem with a huge default limit which will kill the performance of > your site anyhow but isn't enough to kill the entire system? > ...bec

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Rick van Hattem
Hi Mattias, Can you comment on the example I've given where this should not cause any problems and should help pretty much everyone (but just in case, the setting could be *optional*)? If/when an *unsliced* queryset were to reach a certain limit (say, 10,000, but configurable) the system would

Re: Configurable safety options for high performance Django systems

2014-11-23 Thread Mattias Linnap
I thought I'd chime in since I run a Django app with fairly large tables (1M, 30M, 120M rows) on a fairly poor VPS (4GB ram, 2 cores, database and all services on one machine). I just don't see a situation where "safety limits on ORM queries" would improve things. Changing behaviour based on th